Project: TM-β - a home cluster build

Discussion in 'Worklogs' started by Matrixfy, Apr 20, 2017.

  1. Matrixfy

    Matrixfy n00bie

    Messages:
    15
    Joined:
    Apr 9, 2017
    Now that my dual-cpu cube computer is built, I'm free to move on to a new project. Introducing TM-β:

    tmb-cpus.jpg

    Those are Xeon X5650s - 2.66ghz, 6 core/12 thread, Socket 1366

    tmb-cpu2.jpg

    The heatsinks. Supermicro 2U passive, with heatpipes. I was going to go with something bigger, but the boards are... a little special, so I decided not to risk compatibility issues.

    tmb-heatsinks.jpg

    The Ram. 96gb of registered ECC.

    tmb-ram.jpg

    And the boards. Supermicro X8DTT rev 2.

    tmb-motherboards.jpg

    tmb-populated.jpg

    tmb-populated2.jpg

    They take a non-standard 20pin power connection, with only 12v, ground, and 5vsb. I have adapters on order, but I'll hack something together for testing for now.

    TM-β is for Thinking Machine - Beta, or Beowulf. I wanted to build a small cluster to mess around with - a little rendering here, some distributed computing there. I'm going to be custom building a case for these, inspired by various supercomputers:

    A little bit of Thinking Machine's CM-2:

    cm2.jpg

    A little bit of Thinking Machine's CM-5 (Yes, the Jurassic Park one):

    CM5_lg.jpg

    With just a dash of Cray:

    102635318-03-01.jpg

    I've got a lot of other stuff going on, so I won't be updating often, but I hope someone finds this at least a little entertaining.
     
    shadowlord, Wyodiver and DrLobotomy like this.
  2. TsingTao

    TsingTao n00bie

    Messages:
    50
    Joined:
    Nov 22, 2010
    I recognize that board from your 'Workstation 2017' post. Great to see a build thread for it, should be fun to watch.
     
  3. Matrixfy

    Matrixfy n00bie

    Messages:
    15
    Joined:
    Apr 9, 2017
    The first problem to solve is the fact that these motherboards have non-standard 20 pin power connections. They only take 12v and 5v standby, with the rest of the voltages generated on board.

    2PgrogB.png

    I have adapters ordered from ebay, but they're going to take a month to get here. As I've sourced all my parts from ebay, I have to get the motherboards and CPUs tested ASAP, so I took a stab at making my own.

    I have a 4+4 pin extension cable already, so I picked up a 24pin extension cable to get a socket.

    I found out they make ATX pin removing tools, but once again I can't get one in a reasonable amount of time, or for a reasonable price. The alternative was apparently to use staples.

    So far, so good:

    P1010231.JPG

    But, after spending nearly 2 hrs working on the thing, I can't get the last few wires out. I moved on to a donor 20 pin power cable instead:

    P1010232.JPG

    That was much, much easier. Apparently my extension cable is just a piece of junk.

    P1010234.JPG

    And the 4+4 pin extension was likewise simple:

    P1010235.JPG

    So I went back to the 24 pin cable, and spent another hour or two trying to get the last 2 wires out. Then I got desperate.

    P1010236.JPG

    I found out that the tabs holding the pins in weren't fully compressible. There was no way to actually get them out without chopping the thing up. Figures...

    Then I realize that I'm not quite done. I need 8 12v wires, and the ATX 24pin connector only has 2, plus the 4 from the 4+4 connector. That leaves me two short.

    Thankfully, I had a ATX 6pin->8pin adapter laying around, so I pulled that one apart too:

    P1010239.JPG

    With much finger crossing, I hooked the contraption to the motherboard:

    P1010240.JPG

    It's alive!

    P1010241.JPG

    Improbably, all four boards seem to work! Given that I bought the CPUs from two different people, and the motherboards, ram, and heatsinks each from different sellers, I'm shocked that it's gone this well so far.

    Next up is trying to design the case itself.
     
    Last edited: Apr 21, 2017 at 10:32 PM
    Derf and auntjemima like this.
  4. TsingTao

    TsingTao n00bie

    Messages:
    50
    Joined:
    Nov 22, 2010
    Nice work on Frankencable! Good to see it all fired up too.
     
  5. Matrixfy

    Matrixfy n00bie

    Messages:
    15
    Joined:
    Apr 9, 2017
    Thanks :) There was a fair bit of clenching involved when I hit the power for the first time. It didn't help that apparently this board plays dead on a cold start - no heartbeat or power good LED lights, which are otherwise almost always on.

    I said that designing the case was the next thing to do, but I wasn't strictly correct. It occurred to me that there's 2 processors with a TDP of ~100w each, the ram (which actually heats up - not something I've noticed before), and another ~30w for the silly Tylersburg northbridge. That's going to be a lot of heat generation in a 6.75" x 3.5" cross section. Checking power draw and thermals definitely has to come next.

    Since the power load on this board is pretty unique - 100% 12v in operation - I asked the chaps over at Jonnyguru.com for recommendations, trying to keep the cost reasonable. I knew that power draw on a similar board and gold power supply was around 350w or so, so that meant I could get a good quality but relatively low wattage unit.

    The verdict? Seasonics 550-G series Gold rated DC-DC power supply, yours for a bargain price of only $100/ea!

    seasonic.jpg
    (Borrowed this pic - can't find the one I took, and I'm too lazy to go dig up the box)


    And I need 4. Yikes.

    Because it is essentially a 12v power supply and uses DC-DC regulators for 5v and 3.3v, I don't have to worry over cross-loading and minimum load on the other rails.

    I bought one to start. No point wasting money if the boards turned out to be duds.

    We already know the boards boot. Power draw at the wall, with the gold rated supply, turns out to be bang on 350w. That means I should be able to run all 4 off the same circuit - just.

    Thermals have been a bit more challenging. 350w at ~88% efficiency gives about 308w actually being used.

    Before getting the boards, in my head I'd imagined a box with a pair of 120mm fans on each end. It turns out the motherboard is quite a bit smaller than expected, and 2x80mm fans fits perfectly across each end.

    Something like this:

    P1010242.JPG

    Air intake is on the left, and exhaust on the right.

    So I mocked up an exterior shroud to see what would happen.

    P1010243.JPG

    P1010244.JPG

    You may notice there's no video cable connected there. There's only an ethernet cable being routed out of the bottom. As this is a server style board, it has IPMI (IP Management Interface?) built in, which gives IP access to everything from the power switch, and sensor data to video output. Among other things, it's a network KVM that even works for the BIOS.

    Inexplicably, Supermicro's reporting tools do not show cpu temp. Seemingly it's just a high/low value, so I have to boot into an OS to get actual temps.

    ipmi.jpg

    I fired up Prime95 and watched the 2nd CPU - the one farthest from the intake at the back - hits 80c or so, and the chipset (gray heatsink on the right) doesn't do much better.

    Couldn't get it working very well with 80mm fans, so I tried 120mm. The first setup that provided acceptable (sub-70c) CPU temps looks something like this:

    P1010246.JPG
    P1010245.JPG

    Ridiculous. But it worked.

    I decided to go back and try the 80mm fans. I dug up some old 60mm*25mm fans to add to the heatsinks (Does anyone remember the days of Delta Black label screamers? Thankfully I didn't have to resort to those...)

    P1010247.JPG

    The 60mms aren't attached. I'll have to find some creative way of keeping them in place, but they do sit there perfectly, which is good enough for testing.

    And success! Prime95 for an hour didn't get over 65c on the problem CPU, and under 60 for the one on the left. The chipset even stayed under 70c, which isn't terrible either.

    So I"m going to need 16 80mm fans, and 8 60mm ones. Here I thought I was done spending money on this project.

    Come to think of it, I don't have the removable HD bays or drives yet, either.

    And with that success, I'm off to drop another $300 on power supplies...
     
  6. TsingTao

    TsingTao n00bie

    Messages:
    50
    Joined:
    Nov 22, 2010
    Ahh, the good old days. I still have a few in a box that I just can't get rid of...every couple years I'll run across them and fire one up. Such noisy memories.

    Haha, famous last words. ;)

    At least you're not off to spend $2000 on water cooling parts?
     
  7. Matrixfy

    Matrixfy n00bie

    Messages:
    15
    Joined:
    Apr 9, 2017
    Thankfully, no.

    Though I just found out that the 4-bay hot swap bay I want will be over $100 too...


    Now that I know that my thermal solution should more or less work, the next step is actually design the case. I'm a huge fan of blinkenlights, and the supercomputer in Jurassic Park made a big impression on me as a kid. Ever since, I've wanted to build something just like it but it's been expensive, difficult, and with little ability to make the lights useful in any way.

    Now, though, LED arrays can be had cheap from ebay, and driving them with an arduino is relatively easy. Even better, you can actually tie the leds to useful data like cpu load. Even the computers are relatively cheap - Dual CPU, 6c/12t boards for ~$400ca ea. Four of those nets me 48 cores/96 threads at ~3ghz under boost. The stage for my own mini Thinking Machine is set!

    It was the form factor of the boards that convinced me to give it a go. Each motherboard is about 6.75" wide, and 16" or so long - much more interesting than the typical ATX square. I'll be mounting the boards vertically, each in it's own shroud, with intake fans on the bottom, and exhaust at the top.

    Tmb-case.jpg

    Hard drives go in a 4-bay hot swap bay in the front, along with power supplies and cooling intakes to feed the computing "chimneys". The 4 compute units get arranged in a box pattern on top, somewhat like the CM-2, with the LED blinkenlights (with one 8x8 LED array/cpu) goes in the middle, like the CM-5.

    Tmb-case-back.jpg

    The top is removable, and is there to cover the fan outlets and the system for securing the compute units to the frame.

    Tmb-case-exploded.jpg

    That's the plan, so far. I probably won't be posting again for a while, as there's likely to be very little progress over the next few weeks - not only to I have to work out exactly how I'm going to build this thing, but I have to finish turning one side of my garage into a shop to do it in.

    This is only a very rough idea of what I want, and is all very much subject to change. The arduino side of things hasn't even been started yet, and I'm still working out what I want for materials and finish. I'll also probably end up ordering things like the fasteners off of eBay, which means an automatic 1-month delay for delivery.

    Sorry for the ugly 3d. I'm not fond of Sketchup, but it was most convenient thing for some quick and dirty modelling.
     
  8. spugm1r3

    spugm1r3 Gawd

    Messages:
    815
    Joined:
    Sep 28, 2012
    Personally I like Sketchup. Your drawings get the idea across.

    I like the concept, but I have one concern. Those server heatsinks are some heavy duty hunks of metal. I've seen claims of board warping on consumer mobos and, while the server boards mounting screws are closer together, I still have to wonder about hanging them vertically.

    Cheers on the concept, I look forward to seeing how this turns out.
     
  9. Matrixfy

    Matrixfy n00bie

    Messages:
    15
    Joined:
    Apr 9, 2017
    I find sketchup's controls clunky, and the auto snap seems to be erratic at best.

    Thankfully, the heatsinks I have are fairly light. All the weight in them that's there is in the base plate, directly on the socket, so there isn't any additional torque on the board. I'm far more comfortable hanging these heatsinks vertically than I am something like those ridiculously over-engineered Noctua monsters.

    Thanks!
     
    spugm1r3 likes this.