• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Quiet Cluster Cooling

M

Minotaar

Weaksauce
Joined
Mar 21, 2004
Messages
97
Im a grad student preparing plans for my prof for a new 50-100 node duallie cluster in our lab. Currently, we've got a 16 node 1900 MP cluster, each in a mini chassis. Its LOUD. CRAZY loud.

Yes, there is a possibility that we can put the new cluster in another building, but that sets up administration and accessibility issues. For the sake of this discussion, lets assume this cluster isnt going to be in a machine room. But we would ALSO like to keep the labspace holding the cluster as useable as possible: Unfortunately, with 200 cpu fans and 400 case fans whirlling, we'll all be deaf as Kiss. So heres my problem:

I would like to get your input on ways to QUIETLY cool 100 rackmounted duallie boxes. Money is not really an issue; Im just askin for ideas.

In particular, I've been trying to see if it is possible to centrally watercool the cluster, by running all the waterstreams through one big-a$$ exchanger. Has anyone got any experience with this? One of the problems would be the pump: 100 half inch tubes moving water off CPUs is a lot of water per second. It would have to be a hardcore pump.

There are also failure issues: if the pump dies, we dont want to lose 100 machines. Is there a way to get a backup pump to start up if water pressure drops? Are there waterblocks that can dissipate enough heat with no water that the CPU doesnt die before auto-shutoff?

A) how would you watercool rackmounted 200 opterons or 200 MPs? And would it even be that quiet? I think it could be made quiet with central cooling (i.e. giant exchanger, 1 giant pump)

or

B) how would you cool it otherwise?

Constraint: No downclocking/volting
 
Watercooling:

For getting the heat out- get an aquarium chiller (or ten!?)-

http://www.islandaquatics.com/product_detail.php?category_id=15&product_id=151

And use manifolds: http://www.mcmaster.com/asp/DisplCt...9426546959&ScreenWidth=1280&McMMainWidth=1068

http://www.mcmaster.com/

You would spend ALOT of money really quick..

Why not just get a cabinet with doors and panels, and load up on dynamat- I assumed they are rackmount?

http://www.rackmountpro.com/productpage.cfm?prodid=1896

http://dynamat.com/
 
you should go read this article parts 1, 2 and 3. it's kinda like what you wanna do but on a smaller scale
 
I like the idea of using an actual chiller for a simple reason- it'll be tough to get enough radiant capacity for 100 duallies without taking over the whole room with radiators out of 1 TON pickups.
 
but at about $900 a pop you can get vapor phase cooling.... might be over kill tho... and they are damn near silent
but then that is like.. hmmm 180G lol thats alot...
link

you could also buy air conditioned racks no?
 
vapor phase cooling solutions are not unreasonable, but we cannot simply buy 100 vapochill kits and throw em in there.

A) it wont fit
B) even if it does fit, how do you get all the heat off the exchangers? Giant fan?

Vapor phase cooling would work if we could customize a system with a reservoir for the cool fluid, and a pump/manifold system that distributed the fluid to the CPUs, and recollected the vapor back to the compressor. Then used a large exchanger to cool the compressor.

It would be more effective than water cooling, but it would also be harder to set up a coolent distribution system/vapor collection system that centrally handles 200 cpus (as opposed to just one)

Not against it, but unfortunately we'd have to mod the vapochill kit, and by mod I mean totally replace its components with bigger stronger versions of each, and then machine a mechanically sound manifold that will be ok at the operating temps.
 
If youre gonna cool 200 CPU's and worried about the pumps failing, think firetruck. They have 3 pumps for each hose coming out of the truck, on a truck with one hose they could lose 2 pumps and still be pumping water out of the hose. Triple redundancy. It's like RAID but for water-cooling!
 
actually what im doing right now is building my own vapor chiller... but im using it to cool distiled water... if you did the water cooling, you could have -20c liquid cooled systems.
 
dont forget.. doing any type of phase change or chilling would mean insulation on the sockets and tubing.. waaaaaay too much trouble.
 
Most of the models people have mentioned tend to have the same general structure. They have a big loop, with the CPUs generating heat, the water coming over to a reservoir, and the pump pushing the water through a radiator, and back to the CPUs.

i.e.:

cpus-> reservoir -> pump-> cooling-> cpus

This generates a long serial chain which will be harder to pump. What about having a disjoint system that operates with a reservoir in the middle:

Cpus heat water -> reservoir -> pump1 -> Cpus
+
reservoir-> pump2 -> cooling -> reservoir

One of the issues that the The airspirit Watercooled Rackmount Project had was insufficient pump strength - and this makes sense, bc the pump had to move water through the radiator, to the machines, and back, in a big serial loop. However, there could be two pumps, and the water is dumped back into the reservoir by both systems. A simple mixer in the reservoir ensures that the cooled water is mixing into the reservoir well.

This decouples the cooling system from the heatware, and decreases pump load. Note also that the cooling loop could turn on only when water temp rises above some amount, perhaps making it even more silent.
 
I guess I wasn't clear on my suggestion- You'll burn up a hundred really nice machines really quick with a typical homebrew system. It's needs to be well thought out; and seem as though you're in a university environment (grad school, like me) you might spend some time in the other engineering departments before you just 'wing it'. You'd be surprised how helpful and knowledgeable your colleagues are.

Starting at the reservoir-

Have a separate loop from a large reservoir to independent large chillers- these will maintain the temperature of the outlet of the reservoir at say 70 degrees Fahrenheit- won't overstress any part of the chilling chain, these are designed to maintain a 30 degree tank in an ambient room (lobster?). And with your DAQ system (see below) you could monitor the dew point and set the desired temp accordingly. Well- this gets into a control issue, check the end of the reply..

From the reservoir, with several pumps (not aquarium pumps- real pumps with check valves to prevent backflow in the event of failure) running into a main copper supply line (4" etc.) that is branched off into 20 1/2" NPT outlets using stepped copper fittings. On each one of these 20 outlets, fit a 1 port 1/2" NPT in / 5 x 3/8" NPT out brass manifold servicing 5 machines. At each port, install standard machined brass barbed fittings- McMaster sells some with extended barbs (4 instead of 2) to allow zip ties (good suggestion to avoid leaks in a 100 unit cluster..). Inside the machine branch out to the two processors, and that's it- and come right back out. No VPU's, HDD's or anything to worry about.

I would imagine these are not 1U machines- and with power protection and other gear that goes along with a 100 unit cluster, you wouldn't have more than 20 machines in a rack (ideally 10 machine- 2 manifolds in and 2 out; won't have to get 90 degree fittings for the waterblocks unlike the 2U machines). So 4 manifolds per 10 machines.

On the output side, use the same sort of manifolds and hook them to a big 'sewer pipe' draining to the reservoir allowing minimal back pressure that would cause disproportionate flow, but positioned above the intake line to allow the system to 'self bleed'.

In the reservoir, construct baffles the would allow the hot water from the machines to be drawn in by the chillers in stages to allow the coldest water to be at the intakes.

I would expand further, but that is the basic idea.


---

Unfortunately, I've discovered HDD's to be the biggest source of noise- this is where the dyna-matted racks would reign.

But interestingly enough you can get HDD water blocks (I own 10 http://koolance.com) which via their thermal compound and block, absorb a GREAT DEAL of noise. But they're about $50 each; and you're probably running 3 drives per machine in RAID 5 (what I would run in a system like that- with NAS mass storage) that's roughly $61,000 for all of those drives just for the waterblocks; not to mention I at least would be long out of college before Koolance would be able to make 300 of those blocks; it took all in all 6 months to get 10. Doubt you can fit that in your budget; or at least explain it with a straight face. If you can- where are you studying? I need to transfer.

But I promise it would get prohibitively expensive- and I'm afraid no matter how big the budget- without some really loose pockets, water-cooling a 100 unit cluster will become prohibitively expensive.

Well, with just what I have entertained, not including the actual machines, racks, and man-hours..

200 Swiftech water blocks (best out there- engineer's advice- 100 units.. don't use a water block with an acrylic top and a rubber gasket just waiting to fail. looks neat but acrylic cracks and will melt without water..) 200 x $53 (plus 6 months to get them) - $10,600

40 [1x1/2" / 5x3/8"] brass manifolds -
$2,610

Untold Brass fittings and pipe $1,500 est. (most likely much more..)


And unfortunately, here's where it gets a little ambiguous-

Say 300W per machine (WARNING: not at peak for a dually machine- the one's you're talking about at least - and this is only for 2 processors, does not include additional heat dissipated into the ambient air) - you're producing a little over 100,000 BTU per hour.

I am assuming the ratings on these chillers are BTU/hour- if someone wants to take the energy to call them up and clarify that- that would be great. Their biggest one can handle 21000 "BTU Removal Capacity" whatever that is- it's not a rate.

Anyway- with that assumption, you need at least 5. But unfortunately the biggest one they have available can handle 12000. So you'll need a minimum of 9- not to mention if you plan on running it hard, might need more like 10 or 12.

So 10 at $1,800 each: $18,000

You'd probably want a mixing stage per chiller, and a inlet reserve stage in your tank. 20 gallons a minute per chiller (keep in mind 1.5" I/O on those chillers) - I'd say 60 gallons per mixing stage and an additional 100 gallons in the inlet reservoir- need about a 700 gallon tank. Remember 100 dual processor machines.

Need a 3' W x 8' L x 4' T tank. With 10 baffles (10 mixing stages and an intake stage). Can build a steel frame and line it with stainless. Probably will spend $7,000 on the tank.

Pumps. Need a pump for each chiller. Back at McMaster, can get a decent magnetic drive, low pressure pump that would jive with this application for about $440. (~3.5' head from inlet/outlet, another 5' from pipe and chiller: 1/2HP 1" IN, 3/4" OUT - will need some adapters to 1.5" chiller, but that's all right, slower velocity will give higher contact time with cooler)

$4400

Pumps for the main supply line. This gets tricky.. there is much greater head (think a bunch (well, 100) vinyl lines after the coolant gets out of the large supply line and manifolds.). I'd imagine you'd go with 4 2HP centrifugal pumps, and cycle them based upon demand. 2"I/O. 48GPM @ 48' of head- shutoff at 119'. Ploy housing. $1,600

Now you get into controls, where it gets really expensive. Subtotal thus far:

$45,710

That's for parts. Double it for out the door with labor, tax, shipping, bribes and extortion. (need every bit of it to get away with it)

How do you know if you have a crimped line, or a blockage? Well, you really need to have a flow meter at each machine. And a DAQ (Data Aquisition System) system to monitor it. You can multiplex the channels and add all sorts of gizmos to make it work- but to make it automatic and pretty stupidproof like a single machine W/C setup, it would be out of sight.

For a fully self maintaining system, which is possible, add a zero.

Look around at http://www.omega.com

or National Instruments:

http://www.ni.com

They have the stuff if you have a blank check. And if your team is dropping the cash fo 100 dually rigs, you might have that kind of cash, but a couple $3000 soundproofed racks might do the job alot cheaper.
 
This is going to sound different, but if you hooked up multiple pumps like a pumpblock to pump water reduntantly through all the waterblocks then bury a nice large coil about 4 ft in the ground, you could effectivily cool all those computers without a single fan.
 
umm...for things this size normally you dedicate the whole room and you use something like Liebert units. Basically big air conditioners for Electronic rooms where you need air tight protection for Halron fire suppression. This has gone beyond the home stuff. I can get you an idea of cost because we put in a new wiring room at work last fall.
 
First of all, Thank you for your wonderful contribution, jen4950. Very informative.

Current design is to put crap little 50 gig 5400 rpm drives on each node, and run a fiber channel cable to a SAN switch, which will store approx 1 terabyte of raided hot swappable drives. This will be entirely separate of the whole shebang. I dont anticipate drive heat being an issue.

The issue of "other" dissipated heat is significant; the racks, even though passing much water though the cooling system in a water cooling design, still need to be ventilated to get the rest of the ambient heat out, from air around teh water blocks, to heat from components, etc.

Each node is basically a gigE NIC, mobo/ram, scsi controller, ide drive, power supply and 2 cpus. very stripped.

our computing needs are not strongly dependant on exotic network performance, so we dont need to consider myranet stuff.

Im not sure I follow the water volume calculation. each chiller having its own tank? Ok, that means we need a bigger (industrial?) chiller. Not secondary mixing planes which then buffer into a main tank. Larger water cooling apparati must exist (any ideas?), and with such size are probably more efficient than several aquarium coolers.

Also, regarding the number of machines per rack: I'd like to cling to my misguided beliefs for a few more seconds - why cant they be 1U? 1U duallie boxes exist in current application, with air cooling. I just dont think they can handle filling a whole rack of em without watercooling, and hence this discussion. Would like to hear your thoughts on this.
 
Originally posted by Minotaar
Im not sure I follow the water volume calculation. each chiller having its own tank? Ok, that means we need a bigger (industrial?) chiller. Not secondary mixing planes which then buffer into a main tank. Larger water cooling apparati must exist (any ideas?), and with such size are probably more efficient than several aquarium coolers.

Also, regarding the number of machines per rack: I'd like to cling to my misguided beliefs for a few more seconds - why cant they be 1U? 1U duallie boxes exist in current application, with air cooling. I just dont think they can handle filling a whole rack of em without watercooling, and hence this discussion. Would like to hear your thoughts on this.
Click to expand...

Not their own tank- you can't just hook these up in a line- Ice cubes don't miraculously appear as soon as you set the tray in the freezer.. I'm not getting into thermo here...

Code: 
 ^  ^  ^  ^  ^  ^  ^  ^  ^  ^      - Coolant Chiller I/O
-------------------------------------------
|     |     |     |     |     |           |
|  |  |  |  |  |  |  |  |  |  |  |        |
|  |  |  |  |  |  |  |  |  |  |  |        |
|  |  |  |  |  |  |  |  |  |  |  |        |
|  |  |  |  |  |  |  |  |  |  |  |        |
|  |  |  |  |  |  |  |  |  |  |  |        |
|  |  |  |  |  |  |  |  |  |  |  |        |
|  |     |     |     |     |     |        |  > Chilled out
-------------------------------------------
 ^ 
Hot in

Basically what a cooling tower does in a box and without the tower. These chillers are the only self contained units that I have found; where you can set them up in a room without a plumber, and tearing up the whole place. Otherwise, you are going to have to contract it out, and have a unit outside like a typical HVAC system.

Of course my point was kinda that this route might not be the best option on many levels- first cost.

The 1U issue gets back to how are you going to fit a motherboard, processor, waterblock, retention mechanism, and fittings into 1.75" of case?
 
theres actually a water cooled 1U coming from koolance soon. THey've got a 2U, but a 1U is in the works.

in addition to waterblock, they've also got the exchanger and pump iin there too.
 
You must log in or register to reply here.
Back
Top