Distributed Computing on Raspberry Pi

If you’re into or want to learn the crazy world of containers, I can’t think a better way to do it than raspberry pis. You can build a 4 node kubernetes cluster for peanuts compared to pretty much anything else.
 
These are the results from the Rock Pi X. Can't say I'm incredibly impressed with the performance. This is running a standard Linux Mint install, would it be likely to boost the PPD if I installed a Linux distro without a GUI? If so, any suggestions on what distro? I was thinking of throwing on Ubuntu Server(last time I used it, it didn't come with a GUI by default) and see what happens.


rockpix.PNG
 
These are the results from the Rock Pi X. Can't say I'm incredibly impressed with the performance. This is running a standard Linux Mint install, would it be likely to boost the PPD if I installed a Linux distro without a GUI? If so, any suggestions on what distro? I was thinking of throwing on Ubuntu Server(last time I used it, it didn't come with a GUI by default) and see what happens.


View attachment 305972
Performance looks similar to a raspberry pi 4

4EAC8C8F-CA31-4C3F-90FD-8734CFD6468B.jpeg


I found OS and OS tweaks made more of a difference with Rosetta that WCG OP. Rosetta at times needs more ram than a pi can provide, so shaving 200+mb off of memory footprint can make a decent difference (+5% or so). Now, you’ll probably save some cpu by getting rid of background tasks, but while I can point you in the direction, I’m not 100% sure how portable my raspbian tweaks are to Ubuntu.

how much power is that Rock Pi using for those points?
 
how much power is that Rock Pi using for those points?
4.7-4.8 watts according to my Killawatt. That's with a random USB C phone charger I had sitting around so I have no idea what the efficiency of that power adapter is.
 
4.7-4.8 watts according to my Killawatt. That's with a random USB C phone charger I had sitting around so I have no idea what the efficiency of that power adapter is.
Having given this some thought, that actually is fairly impressive. It's not far off in cost from a 4GB Pi 4, and you don't need to add storage so there's no micro sd cost. Power usage is similar, performance is similar, but you get access to all X86 projects/programs not just ones that have been ported to ARM. I might have to think about building a little cluster with a couple more of those and a 5 port switch just to sit in a corner and run WCG. The performance is low enough I won't be tempted to move them to other projects every time a competition comes up. With that said, the Optiplex 9020 SFF with an i7-4770s I picked up used on eBay a couple years ago cost about the same as one Rock Pi X and probably has similar performance to a few of them combined just with higher power usage. Maybe I should run some WCG with that thing plugged into the Kill a watt before I make any decisions, but that's probably not something for the Raspberry Pi/BOINC SBC thread :) I'll probably still grab a Pi 4 to mess around with too. I've owned a couple of Raspberry Pi's but not anything recent.
 
I think the pi 4 is doing more work for less power than the Rock Pi in WCG OP. I also think you’re onto something in regards to using them in projects where there is no arm port though. The pi 4s are about the same power efficiency as a thread ripper 3990x, but just a much smaller slice. As newer hardware comes out, both the Rock pi and raspberry pi will start to lose out in efficiency though.
 
I think the pi 4 is doing more work for less power than the Rock Pi in WCG OP. I also think you’re onto something in regards to using them in projects where there is no arm port though. The pi 4s are about the same power efficiency as a thread ripper 3990x, but just a much smaller slice. As newer hardware comes out, both the Rock pi and raspberry pi will start to lose out in efficiency though.

I might need to start Kill-a-watt'ing more stuff, now I'm curious. I just tested it and the i7 4770s machine uses about 75w from the wall. Honestly, not bad for a 4 core/8 thread 7 year old processor and an old Dell PSU of questionable efficiency, but not even close to Pi 4 or Rock Pi level. That's with 2 4GB sticks of DDR3 and a solid state drive. Probably should order a Pi 4 though to mess around with as well. When I have time I'll try to install a stripped down Linux distro without a GUI on the Rock Pi X and see what the WCG PPD is like without the extra overhead.
 
I might need to start Kill-a-watt'ing more stuff, now I'm curious. I just tested it and the i7 4770s machine uses about 75w from the wall. Honestly, not bad for a 4 core/8 thread 7 year old processor and an old Dell PSU of questionable efficiency, but not even close to Pi 4 or Rock Pi level. That's with 2 4GB sticks of DDR3 and a solid state drive. Probably should order a Pi 4 though to mess around with as well. When I have time I'll try to install a stripped down Linux distro without a GUI on the Rock Pi X and see what the WCG PPD is like without the extra overhead.
Thanks for the update on the 4770. My guess was the pi 4 would crush haswell in efficiency, but I was only able to verify against core 2 quads, old APUs, and a i5-2500k. I’ve replaced half of my old gear with pi4s at this point, but I have a few systems where a pi isn’t the right option.
 
Finally I have a PXE installed and run one Pi without SD Card.
its a bit geeky but works . Very first time one need to run a regular OS on the Pi to change the boot sequence and allow net boot.
But after that it get the files mapped over network.

On my 8GB with ESXi I run an Ubuntu 64bit Server with dnsmasq etc provinding the storage to the SD-less PI over ethernet. And run BOINC.

pi@raspberrypiXX:/var/log $ df
Filesystem 1K-blocks Used Available Use% Mounted on
192.168.mm.nn:/srv/nfs/rpi4-7a123456 32122688 8190424 22277480 27% /
devtmpfs 1867780 0 1867780 0% /dev
tmpfs 1999876 0 1999876 0% /dev/shm
tmpfs 1999876 8580 1991296 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1999876 0 1999876 0% /sys/fs/cgroup
tmpfs 399972 0 399972 0% /run/user/1000
pi@raspberrypiXX:/var/log $



Is this needed ? No. Is it cool ? Oh yeah. Two more Pi's are on the way without SD Card and they will complete the proof of concept. Hopefully I can just clone the already installed BOINC image direct to the new devices.

What is a best way to ensure uniqueness of each BOINC instance ? As I don't want to reinstall the boinc-client ? there is some hardware id ? Isnt it?
 
Finally I have a PXE installed and run one Pi without SD Card.
its a bit geeky but works . Very first time one need to run a regular OS on the Pi to change the boot sequence and allow net boot.
But after that it get the files mapped over network.

On my 8GB with ESXi I run an Ubuntu 64bit Server with dnsmasq etc provinding the storage to the SD-less PI over ethernet. And run BOINC.




Is this needed ? No. Is it cool ? Oh yeah. Two more Pi's are on the way without SD Card and they will complete the proof of concept. Hopefully I can just clone the already installed BOINC image direct to the new devices.

What is a best way to ensure uniqueness of each BOINC instance ? As I don't want to reinstall the boinc-client ? there is some hardware id ? Isnt it?
Yes, there is an ID tied to each Boinc client. Installation is fast enough I just scripted the installation and project association and didn’t worry about cloning it.

are you using iscsi or nfs for the remote storage?
 
Yes, there is an ID tied to each Boinc client. Installation is fast enough I just scripted the installation and project association and didn’t worry about cloning it.

are you using iscsi or nfs for the remote storage?
NFS ...

first tried with centos but dnsmasq / tftp was not working well. Might have been too old.
 
What is a best way to ensure uniqueness of each BOINC instance ? As I don't want to reinstall the boinc-client ? there is some hardware id ? Isnt it?
For multiple boinc instances/clients, I run the linux command "hostnamectl set-hostname [your instance name]". Below is one example where all the 3 instances are on the same physical machine but differentiate between the host versus boinc2 and boinc3 other than the unique BOINC host ID (580196, 580202, etc). Not sure if this is what you ask.

1607793533936.png
 
I’ve been having issues with centOS myself. Perhaps it’s an OS issue
Unfortunately Redhat/Centos changed their intention and centOS become the development release for RHEL. Centos 8 got an earlier EOL :-(
not sure if they will have ARM64 afterwards. Time to divest
 
Tried FAH on ESXi on the Pi ; but the CPU is not properly detected. Works well with Ubuntu 64Bit as VM running 32-bit BOINC on this ESXi. A second dedicated Ubuntu is now driving 5 Pi‘s via BOOTP / TFTP / NFS. No issues yet. The six’th PI will go no-SD-card mode too.
Pain is that the default boot sequence of the Pi is only SD card ; not NetBoot or USB. Still need first to start a regular OS and update the EEPROM. But then based on the MAC address each PI find its own file system on the ESXi. Full lights on the network hub driving that mini cluster.

A4166A4C-47BE-4DEE-8CFE-7A8A4A409BBE.jpeg
There seems some CPU cycles idle on that “server” ...
 
Everytime I read EEPROM I still remember the times I put the EPROM in a UV-Light box to erase them; like the 27C256. [H]eck I even find myself contemplating to buy some little raster board and burn some LEDs.
 
Settled for a new rack and power supply

the cooling is quite good with those big fans; much better compared the little one in the simpler cases or racks.

701A5066-ADA2-471E-ACE0-3B64BAA5FA39.jpeg

And better visibility on power consumption
49293FA1-DD7F-4300-A106-8AE0C9B36AB0.jpeg
Running out of USB A to C cables
 
Settled for a new rack and power supply

the cooling is quite good with those big fans; much better compared the little one in the simpler cases or racks.

View attachment 310870

And better visibility on power consumption
View attachment 310871
Running out of USB A to C cables
Is that a 12 stack pi frame, or can you attach a 8 with a 4 stack? Have you done any power optimizations on the pis?
 
Is that a 12 stack pi frame, or can you attach a 8 with a 4 stack? Have you done any power optimizations on the pis?
It’s a 12slot pi frame, half used. And not yet any optimization done like turn off wifi etc. but with 1A under full computational load I’m ok with already. I saw the suggestions made earlier in this thread and might apply some.

right now the 6 PIs draw 33W (1x8GB, 3x4GB, 2x2GB). My server PI has it own power supply with 4A.
 
If you’re into or want to learn the crazy world of containers, I can’t think a better way to do it than raspberry pis. You can build a 4 node kubernetes cluster for peanuts compared to pretty much anything else.
I might change my mind and will start learning about kubernets. Or another way for better software distribution on the cluster. It’s a bit cumbersome to make sure the bunch of pi run the same software and config. and to avoid excessive scripting. And still need to SSH in anyway.

like installing the node_exporter for prometheus/Grafana is pain in the butt on those loosely couples pi.
 
I might change my mind and will start learning about kubernets. Or another way for better software distribution on the cluster. It’s a bit cumbersome to make sure the bunch of pi run the same software and config. and to avoid excessive scripting. And still need to SSH in anyway.

like installing the node_exporter for prometheus/Grafana is pain in the butt on those loosely couples pi.
Key exchange and remote shell commands not working out well?
 
Key exchange and remote shell commands not working out well?
nah, it would work once setup. no doubts. I also just learned that I could use systemctl remotely. Which is actually useful to make the scripts.

Any good hint for centralized config management ? I'm just youtubing on kubernets as we speak.
 
Depends on what you mean by config management. You can get pretty deep with automated installations and package management with salt (or ansible, or chef, or specific to kubernetes helm ...) but for the scale of a few raspberry pis, a nfs share, some shell scripts, and key exchanges is really the easiest way to handle it.
 
Meanwhile 10 Pi crunching, one Pi as NetBoot server and Grafana monitor.

I also try to have Ubuntu 64Bit via NetBoot which works kind of, but struggle with system upgrades. Stuck in some snaps. Also I thing it run slower compared to Rasbian lite (32) when looking on WCG, somehow one hour longer or a bit more.

maybe because I run a 32bit App on 64bit OS ? Not sure yet.

saw some NanoPi RS4 the other day, a nice little ARM board with two NIC. Contemplating getting one of those as router / NetBoot server for the Pi-crowd to get the NetBoot traffic of my main pipes.
 
Last edited:
Just got my 5950x running WCG. Time to find out which is more points per watt - 5950 or Raspberry Pi.
 
New to the world of distributed computing...What are some potential applications for this?
Medical research, such as Covid antivirals and cancer treatments, physics and astronomy like mapping the Milky Way, Math such as finding prime numbers, and more.
 
Just got my 5950x running WCG. Time to find out which is more points per watt - 5950 or Raspberry Pi.
Got my first set of numbers in. Stock and with PBO enabled, Pis destroyed the power efficiency of the 5950. With PBO enabled and set to motherboard the 5950 whole system pulls 301-306 watts Roughly 68 pis worth of power but “only” 240,682 average points per day over 3 days, which is roughly 40 pis worth of points.

eco mode on the 5950x is a whole different ball of wax though. I only have 1 day or results with the 5950 on Eco, but it kicked out 216,410 points with yields almost exactly 36 times the points per day of a raspberry pi 4, while using 146 watts for the whole system. That around 35 times the power of a power optimized 2GB pi 4, making the eco mode 5950x slightly more power efficient than the pi4. I’m guessing that I got a “good” day on the first day of eco results, and the 3 day average will drop making the pis neck and neck with the 5950x. Starting with zen 4 I don’t think it will be a competition anymore, and the pis will probably fall behind all models of Ryzen 6000 series.

One possible area of gain for the Pis, would be to move to power over Ethernet with a high efficiency switch (92-94% efficient range). If anyone knows much about the PoE switch market, I would love to have some discussions.
 
Last edited:
I made a measurements setup based on the ESP8266 controller and INA219 digital current/voltage sensor.

Tested with some resistors, down to 22Ohm and the measurements were inline with expected Ohm's law results. Safe to assume an accurate 0.01W and 0.01A resolution.

I'll be editing this post and adding more measurements as I go along.

1. Default config as provided by Raspberry PI OS, all packages updated.
No tweaks.
No peripherals.
32GB microSD card.
Cooling is a 60mm slow-moving fan, but powered independently.
OS
: Linux littlepigboy 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux (4GB version)

Idling, boot to console, wired 1gbps ethernet, no wifi
0.49A 2.45W

Idling, boot to console, disconnected ethernet, no wifi
0.38A 1.92W

Device off, ethernet connected.
0.34A 1.73W

Device off, ethernet disconnected
0.24A 1.20W

Idling, boot to console, ethernet+wifi
0.49A 2.45W (spikes when WiFi in use: 0.53A 2.63W)

Idling, boot to console, wifi, but no ethernet
0.38A 1.93W

Idling with GUI, ethernet, no wifi
0.50A 2.52W

Full load (4 Einstein@home tasks, GUI, VNC session)
0.95A 4.74W (a 60-second average)

Full load (4 Einstein@home tasks, no GUI, SSH session)
0.95A 4.70W (a 60-second average)

Idling, console, continuous write from /dev/zero to a file on the microSD. (approx. 22 MB/s)
0.62A 3.08W (60 second average)

Idling, console, Ethernet, continuous write from /dev/zero to a NFS share on another computer (approx 50 MB/s)
0.70A 3.50W (60 second average)

Idling, console, WiFi, continuous write from /dev/zero to a NFS share on another computer (approx 3.3 MB/s)
0.62A 3.08W (60 second average) (note appalling transfer rate compared to wired)

2. The tweaks (as outlined by Endgame in the OP and other guys in this thread)
NOTE: each tweak tested separately;
NOTE: baseline is "
Idling, boot to console, wired 1gbps ethernet, no wifi, 0.49A 2.45W

2.1 Ethernet speed reduced to 100mbps ( sudo ethtool -s eth0 speed 100 duplex full )
0.41A 2.08W (0.08A 0.37W less)
(NOTE: dropping all the way to 10mbps makes things worse - 0.43A, 2.15W)

2.2 Disabling power to HDMI via 'tvservice --off'
0.47A 2.36W (0.02A 0.09W less)

2.3 Disabling power LED ( echo 0 > /sys/class/leds/led1/brightness )
0.49A 2.45W (No noticeable improvement)

2.4 Setting max_framebuffers=0 in /boot/config.txt
0.49A 2.45W (No noticeable improvement)

2.5 Commenting out dtoverlay=vc4-fkms-v3d in /boot/config.txt
0.50A 2.50W (Yes, worse. Outliers will be revisited later on)

2.6 Disabling audio via dtparam=audio=off in /boot/config.txt
0.49A 2.45W (No noticeable improvement)

2.7 Disabling Ethernet LEDs (add lines 'dtparam=eth_led0=4' and 'dtparam=eth_led1=4' to /boot/config.txt)
0.48A 2.44W (0.01A 0.01W less)

2.8 Powering off both USB hubs via "sudo uhubctl -p all -a 0"
0.49A 2.46W (No improvement, even though `uhubctl -p all` shows hubs as disabled)

2.9 Disabling WiFi by adding 'dtoverlay=disable-wifi' to /boot/config.txt
0.47A 2.36W (0.02A 0.10W less - shaves more than just rfkill)

2.10 Disabling Bluetooth by adding 'dtoverlay=disable-bt' to /boot/config.txt
0.47A 2.37W (0.02A 0.09W less - shaves more than just rfkill)

2.11 Adding 'gpu_mem=16' to /boot/config.txt
0.49A 2.45W (No improvement)

2.12 Disabling UART ('sudo systemctl disable hciuart' ;requires reboot, persistent)
0.47A 2.38W (0.02A 0.07W less)

2.13 Adding dvfs=1 to config.txt (dynamic frequency&voltage scaling, default is 2-no undervoltage for GUI installs)
0.48A 2.45W (0.01A 0.02W less)
(NOTE - noticeable positive effect during full load, will investigate)


2.14 Attempt at shedding the USB subsystem via "sudo uhubctl -p all -a 0" and "sudo echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove"
0.49A 2.45W (No improvement)

2.15 Explicit manual undervolt (over_voltage=-4 in /boot/config.txt, dvfs disabled)
0.49A 2.45W (negligible, occasional drops to 0.48A 2.43W, but on a 60 second average it barely registers)
(NOTE - 0.02A 0.10W drop during full load w/Einstein@home)


3. Trying out multiple tweaks at a time
Idling, no GUI, Ethernet. Applied tweaks 2.1, 2.2, 2.7, 2.8, 2.9, 2.10, 2.12, 2.13, 2.15
0.36A 1.84W (60 second average) (0.13A 0.61W less)

Full load (4 Einstein@home tasks, no GUI, SSH session)
Applied tweaks 2.1, 2.2, 2.7, 2.8, 2.9, 2.10, 2.12, 2.13, 2.15

0.78A 3.87W (60 second average) (0.17A 0.83W less)
 

Attachments

  • rasppi4 power mon.png
    rasppi4 power mon.png
    279.2 KB · Views: 0
  • WP_20210523_21_11_28_Pro.jpg
    WP_20210523_21_11_28_Pro.jpg
    322.1 KB · Views: 0
Last edited:
I made a measurements setup based on the ESP8266 controller and INA219 digital current/voltage sensor.

Tested with some resistors, down to 22Ohm and the measurements were inline with expected Ohm's law results. Safe to assume an accurate 0.01W and 0.01A resolution.

Pic of wiring attached.

I'll be editing this post and adding more measurements as I go along.

1. Default config as provided by Raspberry PI OS, all packages updated.
No tweaks yet.
No peripherals.
32GB microSD card.
Cooling is a 60mm slow-moving fan but powered independently.

Linux littlepigboy 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux (4GB version)

Idling, boot to console, wired 1gbps ethernet, no wifi
0.49A 2.45W

Idling, boot to console, disconnected ethernet, no wifi
0.38A 1.92W

Off, ethernet connected.
0.34A 1.73W

Off, ethernet disconnected
0.24A 1.20W

Idling, boot to console, ethernet+wifi
0.49A 2.45W / 0.53A 2.63W ~

Idling, boot to console, wifi, but no ethernet
0.38A 1.93W

Idling with GUI, ethernet, no wifi
0.50A 2.52W

Full load (4 Einstein tasks, "always on" mode, GUI, VNC session)
0.95A 4.74W (a 60-second average)

Full load (4 Einstein tasks, "always on" mode, no GUI, SSH session)
0.95A 4.70W (a 60-second average)

2. The tweaks (as outlined by Endgame in the OP and other guys in this thread)

2.1 Ethernet speed reduced to 100mbps
Idling, boot to console

0.41A 2.08W (0.08A 0.37W less)

Full load (4 Einstein tasks, "always on" mode, no GUI, SSH session)
0.87A 4.3W (0.08A 0.4W less)
This is fantastic!

I’ve got a stack of additional tweaks I’ve been making to my pi build. I’ll quick skim the thread and see if I’ve already posted them. If not, I’ll go ahead and add a reply with the extra modifications, if you’re feeling in the mood to test with tweaking.

post 82 has what to do, but no details. I can provide shell commands to do all those tasks if they would be useful to anyone.
 
Last edited:
This is fantastic!

I’ve got a stack of additional tweaks I’ve been making to my pi build. I’ll quick skim the thread and see if I’ve already posted them. If not, I’ll go ahead and add a reply with the extra modifications, if you’re feeling in the mood to test with tweaking.

post 82 has what to do, but no details. I can provide shell commands to do all those tasks if they would be useful to anyone.
Feel free to drop anything you'd like tested! I'll be playing around with this for a few days. I'll be focusing mostly on power consumption and not performance/memory savings, though, as there'd be a host of additional variables.
Also, no idea yet about the GPU part... can the VC even compute in any of the DC projects?
 
I've just started getting into my Pi 4, and it's a neat little machine. Incredibly efficient. I benched it (of course I did), and it seems to have around 1/10 the processing speed of a 5960X and 1/4 the speed of a 4690K. It's about equal to an AMD A10-7300 mobile APU (on CPU stuff, not GPU stuff). Not bad for 5 watts or so.
 
Feel free to drop anything you'd like tested! I'll be playing around with this for a few days. I'll be focusing mostly on power consumption and not performance/memory savings, though, as there'd be a host of additional variables.
Also, no idea yet about the GPU part... can the VC even compute in any of the DC projects?
I’m not aware of any projects that use the pi video card.

is there any power savings by dropping to 10mbit?
 
I’m not aware of any projects that use the pi video card.

is there any power savings by dropping to 10mbit?

The GPU in my Pi 4 overclocks like crazy if I pump in the voltage. I had the GPU core at over 900MHz with something like 1.05 or 1.1v - can't remember. It was overvoltage level 10-ish. The only problem is that the same voltage feeds the CPU cores and it overheats the VRM in seconds if I try to run something CPU intensive. I was trying to get Quake 3 Arena running the way I like it. That might asking too much.
 
I’m not aware of any projects that use the pi video card.

is there any power savings by dropping to 10mbit?
I'll check that out :) along with the tvservice off setting, LEDs and all that good stuff.
Also underclocking the GPU.
As far as networking is concerned, I think one could get away with something like switching the Ethernet on or off using... GPIOs. I was thinking about making a common UART bus along with simple python listeners to send out simple commands or trigger scripts.
I'll gather some results and will post them tomorrow, probably same time.

Only annoying thing is having to recharge the LiPO I'm using when it drops from the 12.9 or so charged state to around 12.2 which takes no time at all on full load.
 
Last edited:
I’ve toyed with limiting network availability, but it doesn’t work in my case since I’m using Network storage. I also have heard, but not tested, that writing to flash is higher energy use than network.
 
I’ve toyed with limiting network availability, but it doesn’t work in my case since I’m using Network storage. I also have heard, but not tested, that writing to flash is higher energy use than network.
Writing from /dev/zero to a file on the microSD gives me 0.62A 3.08W (so around 0.5W). I'll add it to the results list. Just a quickie lol.
 
Writing from /dev/zero to a file on the microSD gives me 0.62A 3.08W (so around 0.5W). I'll add it to the results list. Just a quickie lol.
A fair comparison would probably be to also write to a network share from /dev/zero.
 
A fair comparison would probably be to also write to a network share from /dev/zero.
Did some more checks, including writing to NFS. Turned out more demanding than writing to microSD, but do note the transfer speed difference.
 
Back
Top