Distributed Computing on Raspberry Pi

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
If you’re into or want to learn the crazy world of containers, I can’t think a better way to do it than raspberry pis. You can build a 4 node kubernetes cluster for peanuts compared to pretty much anything else.
 

Icecold

Weaksauce
Joined
Jul 21, 2013
Messages
118
These are the results from the Rock Pi X. Can't say I'm incredibly impressed with the performance. This is running a standard Linux Mint install, would it be likely to boost the PPD if I installed a Linux distro without a GUI? If so, any suggestions on what distro? I was thinking of throwing on Ubuntu Server(last time I used it, it didn't come with a GUI by default) and see what happens.


rockpix.PNG
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
These are the results from the Rock Pi X. Can't say I'm incredibly impressed with the performance. This is running a standard Linux Mint install, would it be likely to boost the PPD if I installed a Linux distro without a GUI? If so, any suggestions on what distro? I was thinking of throwing on Ubuntu Server(last time I used it, it didn't come with a GUI by default) and see what happens.


View attachment 305972
Performance looks similar to a raspberry pi 4

4EAC8C8F-CA31-4C3F-90FD-8734CFD6468B.jpeg


I found OS and OS tweaks made more of a difference with Rosetta that WCG OP. Rosetta at times needs more ram than a pi can provide, so shaving 200+mb off of memory footprint can make a decent difference (+5% or so). Now, you’ll probably save some cpu by getting rid of background tasks, but while I can point you in the direction, I’m not 100% sure how portable my raspbian tweaks are to Ubuntu.

how much power is that Rock Pi using for those points?
 

Icecold

Weaksauce
Joined
Jul 21, 2013
Messages
118
how much power is that Rock Pi using for those points?
4.7-4.8 watts according to my Killawatt. That's with a random USB C phone charger I had sitting around so I have no idea what the efficiency of that power adapter is.
 

Icecold

Weaksauce
Joined
Jul 21, 2013
Messages
118
4.7-4.8 watts according to my Killawatt. That's with a random USB C phone charger I had sitting around so I have no idea what the efficiency of that power adapter is.
Having given this some thought, that actually is fairly impressive. It's not far off in cost from a 4GB Pi 4, and you don't need to add storage so there's no micro sd cost. Power usage is similar, performance is similar, but you get access to all X86 projects/programs not just ones that have been ported to ARM. I might have to think about building a little cluster with a couple more of those and a 5 port switch just to sit in a corner and run WCG. The performance is low enough I won't be tempted to move them to other projects every time a competition comes up. With that said, the Optiplex 9020 SFF with an i7-4770s I picked up used on eBay a couple years ago cost about the same as one Rock Pi X and probably has similar performance to a few of them combined just with higher power usage. Maybe I should run some WCG with that thing plugged into the Kill a watt before I make any decisions, but that's probably not something for the Raspberry Pi/BOINC SBC thread :) I'll probably still grab a Pi 4 to mess around with too. I've owned a couple of Raspberry Pi's but not anything recent.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
I think the pi 4 is doing more work for less power than the Rock Pi in WCG OP. I also think you’re onto something in regards to using them in projects where there is no arm port though. The pi 4s are about the same power efficiency as a thread ripper 3990x, but just a much smaller slice. As newer hardware comes out, both the Rock pi and raspberry pi will start to lose out in efficiency though.
 

Icecold

Weaksauce
Joined
Jul 21, 2013
Messages
118
I think the pi 4 is doing more work for less power than the Rock Pi in WCG OP. I also think you’re onto something in regards to using them in projects where there is no arm port though. The pi 4s are about the same power efficiency as a thread ripper 3990x, but just a much smaller slice. As newer hardware comes out, both the Rock pi and raspberry pi will start to lose out in efficiency though.

I might need to start Kill-a-watt'ing more stuff, now I'm curious. I just tested it and the i7 4770s machine uses about 75w from the wall. Honestly, not bad for a 4 core/8 thread 7 year old processor and an old Dell PSU of questionable efficiency, but not even close to Pi 4 or Rock Pi level. That's with 2 4GB sticks of DDR3 and a solid state drive. Probably should order a Pi 4 though to mess around with as well. When I have time I'll try to install a stripped down Linux distro without a GUI on the Rock Pi X and see what the WCG PPD is like without the extra overhead.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
I might need to start Kill-a-watt'ing more stuff, now I'm curious. I just tested it and the i7 4770s machine uses about 75w from the wall. Honestly, not bad for a 4 core/8 thread 7 year old processor and an old Dell PSU of questionable efficiency, but not even close to Pi 4 or Rock Pi level. That's with 2 4GB sticks of DDR3 and a solid state drive. Probably should order a Pi 4 though to mess around with as well. When I have time I'll try to install a stripped down Linux distro without a GUI on the Rock Pi X and see what the WCG PPD is like without the extra overhead.
Thanks for the update on the 4770. My guess was the pi 4 would crush haswell in efficiency, but I was only able to verify against core 2 quads, old APUs, and a i5-2500k. I’ve replaced half of my old gear with pi4s at this point, but I have a few systems where a pi isn’t the right option.
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Finally I have a PXE installed and run one Pi without SD Card.
its a bit geeky but works . Very first time one need to run a regular OS on the Pi to change the boot sequence and allow net boot.
But after that it get the files mapped over network.

On my 8GB with ESXi I run an Ubuntu 64bit Server with dnsmasq etc provinding the storage to the SD-less PI over ethernet. And run BOINC.

pi@raspberrypiXX:/var/log $ df
Filesystem 1K-blocks Used Available Use% Mounted on
192.168.mm.nn:/srv/nfs/rpi4-7a123456 32122688 8190424 22277480 27% /
devtmpfs 1867780 0 1867780 0% /dev
tmpfs 1999876 0 1999876 0% /dev/shm
tmpfs 1999876 8580 1991296 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1999876 0 1999876 0% /sys/fs/cgroup
tmpfs 399972 0 399972 0% /run/user/1000
pi@raspberrypiXX:/var/log $



Is this needed ? No. Is it cool ? Oh yeah. Two more Pi's are on the way without SD Card and they will complete the proof of concept. Hopefully I can just clone the already installed BOINC image direct to the new devices.

What is a best way to ensure uniqueness of each BOINC instance ? As I don't want to reinstall the boinc-client ? there is some hardware id ? Isnt it?
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
Finally I have a PXE installed and run one Pi without SD Card.
its a bit geeky but works . Very first time one need to run a regular OS on the Pi to change the boot sequence and allow net boot.
But after that it get the files mapped over network.

On my 8GB with ESXi I run an Ubuntu 64bit Server with dnsmasq etc provinding the storage to the SD-less PI over ethernet. And run BOINC.




Is this needed ? No. Is it cool ? Oh yeah. Two more Pi's are on the way without SD Card and they will complete the proof of concept. Hopefully I can just clone the already installed BOINC image direct to the new devices.

What is a best way to ensure uniqueness of each BOINC instance ? As I don't want to reinstall the boinc-client ? there is some hardware id ? Isnt it?
Yes, there is an ID tied to each Boinc client. Installation is fast enough I just scripted the installation and project association and didn’t worry about cloning it.

are you using iscsi or nfs for the remote storage?
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Yes, there is an ID tied to each Boinc client. Installation is fast enough I just scripted the installation and project association and didn’t worry about cloning it.

are you using iscsi or nfs for the remote storage?
NFS ...

first tried with centos but dnsmasq / tftp was not working well. Might have been too old.
 

pututu

[H]ard DCOTM x2
Joined
Dec 27, 2015
Messages
1,988
What is a best way to ensure uniqueness of each BOINC instance ? As I don't want to reinstall the boinc-client ? there is some hardware id ? Isnt it?
For multiple boinc instances/clients, I run the linux command "hostnamectl set-hostname [your instance name]". Below is one example where all the 3 instances are on the same physical machine but differentiate between the host versus boinc2 and boinc3 other than the unique BOINC host ID (580196, 580202, etc). Not sure if this is what you ask.

1607793533936.png
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
I’ve been having issues with centOS myself. Perhaps it’s an OS issue
Unfortunately Redhat/Centos changed their intention and centOS become the development release for RHEL. Centos 8 got an earlier EOL :-(
not sure if they will have ARM64 afterwards. Time to divest
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Tried FAH on ESXi on the Pi ; but the CPU is not properly detected. Works well with Ubuntu 64Bit as VM running 32-bit BOINC on this ESXi. A second dedicated Ubuntu is now driving 5 Pi‘s via BOOTP / TFTP / NFS. No issues yet. The six’th PI will go no-SD-card mode too.
Pain is that the default boot sequence of the Pi is only SD card ; not NetBoot or USB. Still need first to start a regular OS and update the EEPROM. But then based on the MAC address each PI find its own file system on the ESXi. Full lights on the network hub driving that mini cluster.

A4166A4C-47BE-4DEE-8CFE-7A8A4A409BBE.jpeg
There seems some CPU cycles idle on that “server” ...
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Everytime I read EEPROM I still remember the times I put the EPROM in a UV-Light box to erase them; like the 27C256. [H]eck I even find myself contemplating to buy some little raster board and burn some LEDs.
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Settled for a new rack and power supply

the cooling is quite good with those big fans; much better compared the little one in the simpler cases or racks.

701A5066-ADA2-471E-ACE0-3B64BAA5FA39.jpeg

And better visibility on power consumption
49293FA1-DD7F-4300-A106-8AE0C9B36AB0.jpeg
Running out of USB A to C cables
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
Settled for a new rack and power supply

the cooling is quite good with those big fans; much better compared the little one in the simpler cases or racks.

View attachment 310870

And better visibility on power consumption
View attachment 310871
Running out of USB A to C cables
Is that a 12 stack pi frame, or can you attach a 8 with a 4 stack? Have you done any power optimizations on the pis?
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Is that a 12 stack pi frame, or can you attach a 8 with a 4 stack? Have you done any power optimizations on the pis?
It’s a 12slot pi frame, half used. And not yet any optimization done like turn off wifi etc. but with 1A under full computational load I’m ok with already. I saw the suggestions made earlier in this thread and might apply some.

right now the 6 PIs draw 33W (1x8GB, 3x4GB, 2x2GB). My server PI has it own power supply with 4A.
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
If you’re into or want to learn the crazy world of containers, I can’t think a better way to do it than raspberry pis. You can build a 4 node kubernetes cluster for peanuts compared to pretty much anything else.
I might change my mind and will start learning about kubernets. Or another way for better software distribution on the cluster. It’s a bit cumbersome to make sure the bunch of pi run the same software and config. and to avoid excessive scripting. And still need to SSH in anyway.

like installing the node_exporter for prometheus/Grafana is pain in the butt on those loosely couples pi.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
I might change my mind and will start learning about kubernets. Or another way for better software distribution on the cluster. It’s a bit cumbersome to make sure the bunch of pi run the same software and config. and to avoid excessive scripting. And still need to SSH in anyway.

like installing the node_exporter for prometheus/Grafana is pain in the butt on those loosely couples pi.
Key exchange and remote shell commands not working out well?
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Key exchange and remote shell commands not working out well?
nah, it would work once setup. no doubts. I also just learned that I could use systemctl remotely. Which is actually useful to make the scripts.

Any good hint for centralized config management ? I'm just youtubing on kubernets as we speak.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
Depends on what you mean by config management. You can get pretty deep with automated installations and package management with salt (or ansible, or chef, or specific to kubernetes helm ...) but for the scale of a few raspberry pis, a nfs share, some shell scripts, and key exchanges is really the easiest way to handle it.
 

ChristianVirtual

[H]ard DCOTM x3
Joined
Feb 23, 2013
Messages
2,553
Meanwhile 10 Pi crunching, one Pi as NetBoot server and Grafana monitor.

I also try to have Ubuntu 64Bit via NetBoot which works kind of, but struggle with system upgrades. Stuck in some snaps. Also I thing it run slower compared to Rasbian lite (32) when looking on WCG, somehow one hour longer or a bit more.

maybe because I run a 32bit App on 64bit OS ? Not sure yet.

saw some NanoPi RS4 the other day, a nice little ARM board with two NIC. Contemplating getting one of those as router / NetBoot server for the Pi-crowd to get the NetBoot traffic of my main pipes.
 
Last edited:

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
Just got my 5950x running WCG. Time to find out which is more points per watt - 5950 or Raspberry Pi.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
New to the world of distributed computing...What are some potential applications for this?
Medical research, such as Covid antivirals and cancer treatments, physics and astronomy like mapping the Milky Way, Math such as finding prime numbers, and more.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
Just got my 5950x running WCG. Time to find out which is more points per watt - 5950 or Raspberry Pi.
Got my first set of numbers in. Stock and with PBO enabled, Pis destroyed the power efficiency of the 5950. With PBO enabled and set to motherboard the 5950 whole system pulls 301-306 watts Roughly 68 pis worth of power but “only” 240,682 average points per day over 3 days, which is roughly 40 pis worth of points.

eco mode on the 5950x is a whole different ball of wax though. I only have 1 day or results with the 5950 on Eco, but it kicked out 216,410 points with yields almost exactly 36 times the points per day of a raspberry pi 4, while using 146 watts for the whole system. That around 35 times the power of a power optimized 2GB pi 4, making the eco mode 5950x slightly more power efficient than the pi4. I’m guessing that I got a “good” day on the first day of eco results, and the 3 day average will drop making the pis neck and neck with the 5950x. Starting with zen 4 I don’t think it will be a competition anymore, and the pis will probably fall behind all models of Ryzen 6000 series.

One possible area of gain for the Pis, would be to move to power over Ethernet with a high efficiency switch (92-94% efficient range). If anyone knows much about the PoE switch market, I would love to have some discussions.
 

Endgame

Gawd
Joined
Jan 10, 2007
Messages
530
Last edited:
Top