Distributed Computing on Raspberry Pi

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
I did it old school here with a multimeter to measure current draw and using Ohm's Law. (being that I'm an old tech)


First the pi sitting booted up to the desktop in the Pi OS and at idle. Again with mine the CPU is a stock clocks and no hardware is disabled on the pi.

View attachment 258908

Next I loaded Open Pandemics with 4 threads running and the pi under load.

View attachment 258909

Using Ohm's Law (P=E x I) so 5 vdc x 890 milliamps = 4.45W


I've included a video below showing about 12 seconds of it running under load to show how that current draw is varying some, in this as high as 4.8W for a second. Also I did see it touch 1 amp (5W) for a second after it first started running O.P. So in my particular case it's running under load from around 4.4 to peaks of 5W.
View attachment 258904
Fantastic - then my average of 4.25 watts might even be a little high! Good work
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
I had one of the little cooling fans start sounding like a pissed off hornet (the unit that was in my room of course) so I'm looking at a different solution. Should be delivered today via Amazon.
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
I had one of the little cooling fans start sounding like a pissed off hornet (the unit that was in my room of course) so I'm looking at a different solution. Should be delivered today via Amazon.
Yep, those tiny fans don’t last long.

what did you go with?
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,712
So, I'm not terribly interested in doing distributed computing work on my Pi's; however, I did just put together a Pi 4b 4GB as a 'network appliance' replacement for the 3b I had been using. It does piHole and Unifi controller duties for the moment.

Before I put it into service, I did a few weeks of random benchmarking with it overclocked, inside a fanless Geekworm case, and toward the end with a PoE adapter (one with a fan). In the end I was able to get it to top out around 65c at 2GHz CPU and 700MHz GPU.

The eventual goal is to put together a Kubernates cluster of sorts. What I'm hung up on / annoyed by, other than the cost, is that there doesn't seem to be a good solution that integrates enough cooling and PoE into a solid 'solution'.

I'm also interested in trying the USB boot functionality, or perhaps setting up network boot only, as well as setting up networking monitoring clients (i.e. Zabbix) to poll the hardware at some point.


Feel free to tell me to eff off if the side discussion isn't useful for dialing in hardware :)
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
Seems ok but never own one. The manual says accuracy standard class 1.0 meaning error can go up to 2% which is comparable to kill-a-watt.
Order just went in, and I should have my new meter by Thursday
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
So, I'm not terribly interested in doing distributed computing work on my Pi's; however, I did just put together a Pi 4b 4GB as a 'network appliance' replacement for the 3b I had been using. It does piHole and Unifi controller duties for the moment.

Before I put it into service, I did a few weeks of random benchmarking with it overclocked, inside a fanless Geekworm case, and toward the end with a PoE adapter (one with a fan). In the end I was able to get it to top out around 65c at 2GHz CPU and 700MHz GPU.

The eventual goal is to put together a Kubernates cluster of sorts. What I'm hung up on / annoyed by, other than the cost, is that there doesn't seem to be a good solution that integrates enough cooling and PoE into a solid 'solution'.

I'm also interested in trying the USB boot functionality, or perhaps setting up network boot only, as well as setting up networking monitoring clients (i.e. Zabbix) to poll the hardware at some point.


Feel free to tell me to eff off if the side discussion isn't useful for dialing in hardware :)
I would abandon PoE as a solution unless:
1) you have a use case where no other power option is viable
2) your load profile has the pi at 50% or less usage.

Poe hats + per port PoE switch costs end up more than doubling the cost of a 2GB pi 4. I would rather just find a way to run a 3’ usb C cable.

I’ve linked to a kubernetes pi guide over on the Anandtech post on the first post, but I haven’t tried it yet. My next step is to get my pis network booting, and I ran into a complication with my PoS router that it doesn’t support PXE boot. I spun up a vm on my freenas to handle that, but really want to run it from a dedicated NIC.

Finally, pihole barely uses cpu - throw a thread on open pandemics at least :D
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,712
Well, I already have the PoE switch, and with the hats at US$20/ea, it wasn't a big ask to try. Right now my Pi 4b is running with just a network cable attached; previously it was doing so with a keyboard and mouse attached too, overclocked, running CPU benchmarks. The Unifi controller claims it averages around 5w in that config.

And network booting would be nice; I'll have to look into it for other Pi's, just don't necessarily want to do that for this one :)
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
Turned out pretty good. I like it. 👍

picluster.jpg
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
And it looks like you have room for a couple more Pis!

The per unit cost is a little high, but it’s certainly less janky than my current setup:

If someone was just wanting to purchase boards, power, and cooling, its cost isn't that bad for something designed for 4 boards vs getting some other type of case and cooling. Plus I have to say that I was impressed with the quality of the pieces and the amount of included hardware.

Here's one of them showing temps at idle then again after running O.P. for about 5 minutes. Roughly a 10C increase using that solution.

geeekcooling.png
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
If someone was just wanting to purchase boards, power, and cooling, its cost isn't that bad for something designed for 4 boards vs getting some other type of case and cooling. Plus I have to say that I was impressed with the quality of the pieces and the amount of included hardware.

Here's one of them showing temps at idle then again after running O.P. for about 5 minutes. Roughly a 10C increase using that solution.
I can't tell from the picture - is there room for a smallish heatsink on the broadcom chip? It certainly looks like you could put a fairly large heatsink on the top layer, but its harder to tell on the mid layers.

If you really wanted to contain the per pi cost, you could go with the 8 stack tray, which brings the cost down to roughly $8.25 per pi - half the cost of using a flirc case on each pi.
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
Yeah all of my Pis have three copper heatsinks on the main chips.

1594299485625.png


Although it looks a little tight there's still plenty of room on each layer.

1594299699941.png
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
After playing with the power meter a little bit, I have these observations.

1) Anker customer service stated its 6 port power supply is 85% efficient at any load. This appears to be incorrect. 1 pi 4 4GB attached to the charger draws 5.43 watts peak running Rosetta. 5x pi 4 4GB plugged into the same charger draw 24.79 watts peak

2) Rosetta generates a dramatically more variable load than WCG OP. Rosetta will vary by >.75 watts with 1 pi 4, while WCG OP varies by < .1 watt.

3) During load, multiple tweaks seem to make minimal to no difference in power utilization. Disabling USB, hdmi, and audio have no effect. Turning off the LEDs and slowing ethernet to 100mbit seem to be the most reliable way to reduce power.

4) the lowest observed power draw at the meter is 4.516 watts per pi (using the 5 pi setup). This is not a great showing for efficiency - that's 311 Roseta RAC per watt at the current point values, or 243.58 April RAC per watt - this means there are certainly AMD processors that are more efficient per watt.
 

pututu

[H]ard DCOTM x2
Joined
Dec 27, 2015
Messages
1,945
Endgame, from what I read here, I think a node size shrink in the future down to 7nm (comparable to current ryzen 3000) will likely increase the PPD/watt.
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
Endgame, from what I read here, I think a node size shrink in the future down to 7nm (comparable to current ryzen 3000) will likely increase the PPD/watt.
Right - the pi4 is at the 28nm node, which makes it less efficient than 7nm AMDs. However, it’s low Power optimized ARM so it’s base is more efficient than AMD which offsets the manufacturing node.

In the end, it appears that the pi4 is roughly equivalent to the high cpu count Ryzen 3000 line, and will fall behind over time. To keep costs down, the pi foundation is expected to stay at 28nm for another 5 years or so, so we will likely never see a leading node pi, or even a near current node pi.

It’s been a fun project, but there appears to be no great power efficiency gain to be had. :(
 

pututu

[H]ard DCOTM x2
Joined
Dec 27, 2015
Messages
1,945
It’s been a fun project, but there appears to be no great power efficiency gain to be had
Keep in mind that for ryzen 3000 to achieve high efficiency, one would need to under-clock and under-volt the CPU. There is a good article here which I use to optimize my 3950x. Scroll down to the section under "Performance per watt". Most folks run at stock or overclock, so won't see that efficiency gain. Can the pi be under-volted?

Running 3950x at 3.5GHz seems to be the sweet spot.
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
Keep in mind that for ryzen 3000 to achieve high efficiency, one would need to under-clock and under-volt the CPU. There is a good article here which I use to optimize my 3950x. Scroll down to the section under "Performance per watt". Most folks run at stock or overclock, so won't see that efficiency gain. Can the pi be under-volted?

Running 3950x at 3.5GHz seems to be the sweet spot.
Yeah, my gold standard is underclocked and undervolted AMD hosts, such as StefanR5R's (Anandtech forum member) power optimized dual Epyc 7452 (dual 32c/64t Zen 2).

My full power optimizations on the Pi 4s that I'm testing with, in order of effectiveness:
Ethernet reduced to 100mbit
Maximum under volt (0.835v). Exactly how much this under volts the CPU varies, as the stock voltage for Pi 4 seems to vary by board. I've seen stock voltage as low as 0.85v and as high as 0.86v at idle.
disable power LEDs
disable hdmi / audio / usb (no measurable drop in power)
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
Playing around with overclocking again with the power meter (still on Rosetta, which isn't perfect), it seems that on the pis that will run 2000mhz on over_volt=4 (+.1v) pull 6 watts (give or take a little due to Rosetta fluctuations). That is a 31% increase in clockspeed for roughly a 33% increase in power, which is actually really quite good. Unfortunately that extra clockspeed generally only results in about +20% rac in Rosetta, but its worth keeping track of, and may be a bit more linear in a project that doesn't use quite so much ram.
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
It’s been a fun project, but there appears to be no great power efficiency gain to be had. :(

I would make only this point. Currently with the exception of the Pis I cannot even turn on another computer and have it sit at idle and draw only 30 watts, much less 15W. The Pis provide me the ability to make a choice to still work on research using 12 cores, albeit at a slower comparable pace, and draw 25% of the power that my current desktop does sitting here doing no research at all while it moves electrons and generates heat. I have a choice I didn't have before to work on science in a "low power" mode. Thanks to your information about the Pis in D.C. use, and I can add or subtract to that choice 5 watts/4 cores as I please. I appreciate that choice for my computing needs. 👍
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
I would make only this point. Currently with the exception of the Pis I cannot even turn on another computer and have it sit at idle and draw only 30 watts, much less 15W. The Pis provide me the ability to make a choice to still work on research using 12 cores, albeit at a slower comparable pace, and draw 25% of the power that my current desktop does sitting here doing no research at all while it moves electrons and generates heat. I have a choice I didn't have before to work on science in a "low power" mode. Thanks to your information about the Pis in D.C. use, and I can add or subtract to that choice 5 watts/4 cores as I please. I appreciate that choice for my computing needs. 👍
This is very true, and its worth noting that currently a pi 4 is equally efficient to top end processors that are very power tuned. I’m fairly confident that you’ll get better points per watt out of a pi 4 than, say, a Ryzen 3800x, or likely any Intel processors.

Additionally, you can replace older equipment with pi 4s and use dramatically less power. 2x 4gb pi 4s put out About the same amount of points as my core 2quad 9650, while using roughly 110 less watts of power.I think most people running anything haswell or earlier could make a good case for replacing the hardware with a 6 pack of raspberry pis, especially if their project of choice could fit comfortably on the $35 2GB version.

at the same Time, I don’t think building a 100 pi cluster is the right call. Instead of managing 100 pis, just go buy a threadripper 3990x.
 

Gilthanis

[H]ard|DCer of the Year - 2014
Joined
Jan 29, 2006
Messages
8,196
at the same Time, I don’t think building a 100 pi cluster is the right call. Instead of managing 100 pis, just go buy a threadripper 3990x.
This right here holds a lot of value. I had over 30 ARM devices in my phARM at one time. I can tell you it is a PITA to manage all of them and make sure each one is happy and not just sitting there idle or worse. As I progress through adulthood which now includes kids, I must also cut back on time spent hovering over rigs. There is HUGE value to having a lot of eggs in one basket. Massive amounts of cores in one box may very well be worth a premium price at the end of the day. A lot of people know that I'm really frugal and run a lot of old hardware. However, as time has gone by I have moved away from small core counts and merging to fewer rigs with larger core counts. It is still older but a lot less overhead when it comes to my time. I'm always curious about the Pi's but doubt I ever bite. If someone would put out a legit cheap server board that could handle a couple hundred cheap ARM cores, then they would have my attention.
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
This right here holds a lot of value. I had over 30 ARM devices in my phARM at one time. I can tell you it is a PITA to manage all of them and make sure each one is happy and not just sitting there idle or worse. As I progress through adulthood which now includes kids, I must also cut back on time spent hovering over rigs. There is HUGE value to having a lot of eggs in one basket. Massive amounts of cores in one box may very well be worth a premium price at the end of the day. A lot of people know that I'm really frugal and run a lot of old hardware. However, as time has gone by I have moved away from small core counts and merging to fewer rigs with larger core counts. It is still older but a lot less overhead when it comes to my time. I'm always curious about the Pi's but doubt I ever bite. If someone would put out a legit cheap server board that could handle a couple hundred cheap ARM cores, then they would have my attention.
I’ve got 2 small kids of my own, so I get the time crunch. There are management tools that could let you run a very large number of hosts, though learning and setting up something like that is a non trivial task. I really need some Kubernetes exposure for work, so I’ll be able to justify company time to get something working, which can include my own cluster - not everyone will be so lucky.

as for arm boards, Anandtech covered a 32 core development box recently. It is really aimed at needing a development platform for arm servers and used some fairly old arm equipment, but if the platform gets a few updates, it could be the new points per watt platform of choice.
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
If I was going to set up a lot of Pis (still a future possibility) then I'd probably configure them to email or text me once a day so I know it's still alive and connected.
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
If I was going to set up a lot of Pis (still a future possibility) then I'd probably configure them to email or text me once a day so I know it's still alive and connected.
If you go this route, I can suggest different ways of doing this that will be less time intensive for you. Lets say you have 12 pis - if you configure e-mails to announce the pis being alive, you'll have to look for 12 e-mails every day.

Alternatively, super easy, turn off all the power LEDs like I noted earlier in this thread and place the pis somewhere you look every day (I walk past mine on the way to my office every day). Sometimes when dialing in an overclock, a pi crashes, reboots, and gets stuck - when this happens, the power LED turns back on, so you'll have a bright red LED letting you know which one you'll have to unplug and plug back in. On the plus side, I've found the Pi 4s to be super stable with the only real problems coming from using too much ram on Rosetta processes.

The next level beyond just looking for red LEDs would be active monitoring with Nagios, and configure it to e-mail and / or text you if a host goes down. Now you only need to wait for alerts that may only happen every couple of weeks instead of checking daily.

The final level would be a full cloud approach running Boinc in containers and managing them with Kubernetes. When I figure this out, I'll write a guide specific to distributed computing.
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
Another Pi 4 / 4 should arrive today to populate that last layer in the little cluster case. With that filled I'll probably put a pause of buying Pis for a bit.

I'm about to get involved in personal business and dealing with estate matters for the near future. (not the fun kind)
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
The final level would be a full cloud approach running Boinc in containers and managing them with Kubernetes. When I figure this out, I'll write a guide specific to distributed computing.

Or as just popped into my head this morning (at least this works in my current situation) I just look at the device statistics in WCG and see if a particular unit hasn't reported in the last 12 hours.

1594552834235.png


One place to review all action and investigate any slackers. ;)

The 4th Pi came yesterday and is crunching.
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
Or as just popped into my head this morning (at least this works in my current situation) I just look at the device statistics in WCG and see if a particular unit hasn't reported in the last 12 hours.

View attachment 260874

One place to review all action and investigate any slackers. ;)

The 4th Pi came yesterday and is crunching.
Yes, this works well too if you aren’t crazy like me and setup for half a dozen projects :D
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
I picked up another 8GB pi and another 2GB pi. The second 8GB pi will be a Kubernetes Master replica, and the 2GB pi will be a worker node for WCG OP, as well as my test Pi for PXE boot.

While I’m unlikely to move beyond 12 nodes in my distributed computing cluster, I think I’m going to swap out my pi 2 based Pihole for a pi 4. Will probably try to sell off the pi2 for shipping and $10.
 

Gilthanis

[H]ard|DCer of the Year - 2014
Joined
Jan 29, 2006
Messages
8,196
I pictured you more as an Ice Tower kinda guy. ;)

View attachment 261079
I'm too cheap to do one that nice. Plus the foam board option is more enclosed preventing a lot more oops as well as being more compact. Sticking 4 in the space of like 1 is a lot more appealing to me. Less chance of my 3 year old sticking a screwdriver in it...
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
9103F5A7-8F5A-44BF-8BDB-1CB1335D8939.jpeg



power optimized pi 4 2GB running raspbian lite and clocked at 1525mhz. I haven’t totaled it up to get a real average, but looks like its about 6k?
 

Endgame

Limp Gawd
Joined
Jan 10, 2007
Messages
334
I tried fitting my old zalman super flower copper heatsink on my newest pi 4. Nope, not going to happen - it seemed reasonable when I thought about it given how small the Athlon T-Bird die is, but I didn't take into account a Raspberry pi board is essentially the same size as the Athlon Socket A socket. There are just too many surface mount connectors on the pi that have a z height taller than the broadcom chip. I ran through a few different old pentium heatsinks I had on had too, all with the same result. You would really need a heatsink with a small, quarter sized pillar probably 1-1.5" tall to clear all the stuff on the surface of the pi. Even with the right heatsink, the broadcom chip is not centered on the board, so you would have to have a good attachment mechanism as well.
 

The_Heretic

Certified [H]
Joined
Jun 22, 2001
Messages
13,358
Looked back this morning at the status of the 4 Pis and it looks like they've passed their first million points (combined) with Open Pandemics
thumbsup.gif
1,208,827

1598875344325.png
 

metallicafan

[H]ard|DCer of the Month - May 2010
Joined
Mar 30, 2005
Messages
2,201
I've long thought about getting a Pi to tinker around with but have never done so. Just curious about your expereince with the aforementioned FLIRC pi 4 case. Does it provide enough passive cooling to run all four cores on WCG (or other DC projects) 24/7? As I started to search for Pi4 cases and read reviews I saw a lot of mixed thoughts out there about the amount of heat generated and the cooling provided by these cases.
 
Top