Massive Water Cooling Setup Suggestions Welcomed

JJ91284

Gawd
Joined
Mar 28, 2007
Messages
696
I currently have 53 XFX R9 290's and am looking to watercool these cards as they obviously put out lots of heat and are loud. I'll start off by saying I've never water cooled a setup before, but I'm very handy and can do plumbing and electrical. I have my current setup on 2 dedicated 50 amp circuits at 240v.

Currently I'm considering the EK Water block with backing
http://www.frozencpu.com/cat/l3/g/c...cks-EK_Blocks_-_VGA_ATI-EK_R9-290X-Page1.html

I'm thinking of running a 1" supply line that then branches to 3 systems of 4 gpus each. Each set of 4 gpus then would be cooled in series which then would dump back into the output line (heated water) into a 1" tube that runs to the heat exchanger outside.

Input line 1", then splits into 3 separate 1/2" lines which feed 4 gpus in series, which then dumps back into a 1" output line. I'd need to run 4 separate heat exchangers and 8 pipes total for intake and output lines.

The other idea is to run a massive 2.5 inch supply line and output line and right before it gets to my computer rigs would be to branch the supply into 4 smaller 1" lines to feed each gpu set in series. This I think is better as it requires less plumbing. The only down side is if something happens to the main line, all the gpus are going to overheat and hopefully thermally throttle back.

So for pumps, I was thinking of either going with car intercooler pumps since I'm pretty familiar with these. And then using a water to air intercooler or a car radiator to dissipate the heat outside.

Current pumps I'm considering
http://www.frozenboost.com/product_...=1001&osCsid=8cf9adeec98aa03d3b8168c19b20891d


or maybe a Mezeire electric water pump. I'm familiar with these 12v pumps and they work great in high heat automotive environments. I don't have any experience with well water pumps, but I think this may be another solution. Either way the entire cooling loop is about 150 ft long so for the 12v pumps I might have 2 run twin pumps at like 75 ft or so to keep the flow going and cause less stress on the pumps.

Current Heat Exchangers Example
http://www.frozenboost.com/product_...=1026&osCsid=8cf9adeec98aa03d3b8168c19b20891d

or I'll find a big truck radiator. Ideally speaking the radiator would want to be able to dissipate 10,000 watts of heat.


 
Last edited:
Honestly at that cooling requirement your beyond water IMO. The complexity for a loop will give you too many possible points of failure (IMO). However geothermal could be interesting side project..


My personal thought is that if your looking at WCing everything, look at maybe dedicating a new AC unit to the room and just try and control your ambient more so than the cards themselves.
 
Several problems you're facing here hoss.

1) Waterblocks are individually expensive...you're looking at $8000 alone in waterblocks. There's also the problem of finding any retailer that keeps that many in stock

2) You need to plumb them in parallel, due to the massive number of blocks and the restriction of each block....which complicates plumbing quite a bit

3) In parallel, you'll need lots of pumps to keep flow rates up.

4) 10kW is a crapload of heat to sink...which is what you're looking at. That is the kind of thermal load residential home air conditioners barely keep up with. What are you going to sink all that heat into? It has to go somewhere.

5) As already mentioned...hundreds of failure points.

It is less problematic just to build your own refridgerated meat locker.
 
i am in iowa, i will send you some cold

feels like -18F
 
A correction BTW...from what we can gather each 290 pulls 295W which means you're not needing to sink 10kW, you're needing to sink 15.6kW. A difference of only 30% ;)
 
So thats where all the 290's went. Damn LTC miners

Agreeing with everyone else that you're far beyond any kind of "normal" WC loop here, to the point where it would likely be more efficient to go with a dedicated refrigeration steup.
 
My advice would be to create a barrier using plastic sheeting around the rack and duct outside air at the bottom of the rack and hot air at the top of the rack directed to the outside.

I design data centers for a living and that's the most cost effective way to cool any IT equipment.
 
Guess who's about to get raided by the DEA... :eek:

Run one water input line and one outlet line per row of cards to allow running the coolant flow in parallel. These main lines should be tapered, but it's probably not a deal killer... maybe.

Install quick disconnects on both lines. (http://koolance.com/help-quick-disconnect-shutoff-couplings) Space them to correspond with the cards. Using QD are critical as you cannot afford to have to shutdown the whole system to remove one card.

As you get water cooling plates for the cards, install tubing pigtails with the mating quick disconnect. The hose doesn't need to be huge, you're only trying to keep these thing alive, not cool them to death.

during the winter/spring a single very large radiator should sufficient, but plan to add radiators as it warms up. On very hot summer days you might have to run water mist generators on the radiators to gain evaporation effects.

I think it's doable, but not for the faint of heart or those with a thin wallet...
 
As an employee at the IRS, I would not want to be involved in BitCoins or anything like that. Audits are coming down soon and hard, and taxes are going to hurt. I think they want to kill bitcoin tbh.
 
The problem you're likely to face is water cooling or air cooling won't matter in this case. 10kW of heat to remove still goes into the room's air regardless of the type of heatsink you have on the GPUs. What you should focus on first is getting the expelled heat out of that room so the GPU fans (or radiator fans) aren't pulling in hot air in the first place.
 
What does one do with 53 - 290's ? Just wondering.

Print money...

Well you would plumb them parallel with a resivor and you would chill the collection barrel.

Also as for the IRS the us does not control the world as much as that yeah they are going to find a way for their slice to be taken.
my thoughts on a solution are a federal sales tax on everything and dropping income tax. It gets the illegals and visitors takes it out of alot of peoples hands and makes tax Time really easy as there is no more tax returns but this would kill so many jobs IRS downsized h&r block gone. Which I think is the main reason my idea will never happen.
 
Last edited:
16kw = 55k BTU... just get a home AC unit for that one room. one with a 80-100kw/hr rating should give you enough power to play with. thats going to be the most cost effective.

no need to look into walkin fridge/freezers, going below the dewpoint, and having to deal with all that condensation on thousands of $ worth of hardware. 60-70 degrees is fine, you just have to move a lot of energy.
 
A walk in meat locker or room/sub-room sized cooler ordered from a restaurant supply dealer is the simplest solution and probably about the same price or possibly less than all the components for a water cooling setup of this size. Have it coolant/chiller/compressor and everything else setup the same way that restaurants do as well.

I started writing out a plan for a water cooled setup (which would very much need to be in parallel for every 4-8 cards) and it just was not worth the hassle of all the steps and parts.
 
As an employee at the IRS, I would not want to be involved in BitCoins or anything like that. Audits are coming down soon and hard, and taxes are going to hurt. I think they want to kill bitcoin tbh.

An attempt to kill Bitcoin (or any "uncontrolled" digital currency) is absolutely a top priority from the government's perspective. A successful alternative currency that bypasses the central reserve will destroy the US government. It's not a matter of if, but when the gov't goes into full swing btc attack mode.

Hmm... that sounds all paranoid and conspiracy theory like, but it's really just the way the math works out.
 
An attempt to kill Bitcoin (or any "uncontrolled" digital currency) is absolutely a top priority from the government's perspective. A successful alternative currency that bypasses the central reserve will destroy the US government. It's not a matter of if, but when the gov't goes into full swing btc attack mode.

Hmm... that sounds all paranoid and conspiracy theory like, but it's really just the way the math works out.

Why wouldn't the government want to use a currency they can't just print more of?
 
16kw = 55k BTU... just get a home AC unit for that one room. one with a 80-100kw/hr rating should give you enough power to play with. thats going to be the most cost effective.

no need to look into walkin fridge/freezers, going below the dewpoint, and having to deal with all that condensation on thousands of $ worth of hardware. 60-70 degrees is fine, you just have to move a lot of energy.

Having a separate dedicated AC zone with a 5 ton unit would be cheaper if little to no duct work is needed. But then you would still need a whole bunch of directed fans over the cards, motherboards and PSUs to help physically move the localized air over the cards in the room. Each board still needs to deal with the thermal load caused by having 75 watts per GPU passing through it. In the pictures it looks like there are a dozen plus motherboards, so those ancillary components of PSUs, CPUs, MBs, USB sticks/HDDs/SSDs, and fans will toss a good 1-3 more KW worth of heat into the room.

Moreover all that noise from needing to run the fans at 70-80% would not be blocked out by regular home drywall. All of my experiences with datacenters are that they loud and have lots of carefully directed cooling through many sub-assemblies of fans, both in the rooms and in the racks. Even then, those are generally regular server farms rather than the special coolant needs for miniframes stocked to the gills with GPUs running at 100% load 24/7.

A freezer's much lower starting temperature would deal with the directed air flow needs for the MBs and PSUs while the constant outpouring of heat would soon get things above the dew point inside the walk in. At that point the goal would be to keep the freezer's workload at thermal parity to the mining array rather than trying to actually freeze anything. A freezer's thick metal walls will also contain the noise. It looks like the biggest downside is that a power/coolant failure for the freezer would rapidly cause the whole array to overheat and fry itself whereas a dedicated air conditioner would have a longer safe-ish time till thermal overload.
 
Last edited:
well, if one gets something like NZXT G10 coolers then you have the bracket and just need the liquid cooler part, if you are handy with soldering then you could run the lines to a massive water tank (hotwater tank as example) then water goes from tank to cards 1 line in, the other line on return can go to rads of whatever type you need this can go external outside or whatever and if the rad is large enough can be passive.

AC wise this would be wicked $ constant but is the best way to cool them all at once, everything else will be complicated for nothing. creating a boxed in room where the cards are and having the ac central air whatever pump the cold into said room and pump its heat outdoors can be done without much issue.
 
placeboFX is on the money with this for air cooling. Isolate the rack from the rest of the room, force cold air in and hot air out. Constant airflow is key, and you'll need to keep the humidity under watch as well. Too little and the air doesn't conduct heat as well. Too much and you get condensation on cold surfaces that can run down onto exposed parts. That's the approach I'd use.

If you're really intent on water for OC and noise... I'm assuming the former is the primary reason... with the amount of tubing you'd need you'll eventually have to deal with condensation and potential leak problems. One leak on the top of the rack and your farm is done. Same with drips on cold lines. You'll definitely need humidity control. I imagine you're air conditioned anyway, but if you're going to be using an open reservoir at all that's a concern. As for piping, just remember that the longer your pipe run is the lower your pressure is at the end. I'd use two larger flow pipes run lengthwise with the card racks, flowing parallel from cold over each card to a hot return, like this viewed from the top:


| ================= <------ Cold
|| || || || || || || || || --- Inflow Line
X X X X X X X X X --- Cards
|| || || || || || || || || --- Outflow Line
| ================= ------>Hot


The ends of the hot and cold rails would be capped so the flow is forced over the cards. Use 1" to 1.5" pipes for your cold and hot rails for each rack, junction each rack rail into 2" to 3" vertical master pipes. I would feed the master pipes at the center to even the pressure between the rails.

If you're going to use that model pump you might need two with the above setup - one on the hot in and one on the cold out of your radiator. I'd prefer something a little beefier myself for a single pump design. The main reason I say this is you'll have back pressure at the radiator and you'll want to keep the pressure up enough to get good flow at the far end of your rails. More flow = more efficient cooling, so long as you don't put too much pressure on the lines. A little testing is always good before mounting any heat blocks onto the actual cards. I'd build the rig first, give it a shakedown run, then after you're satisfied it's working without trouble and your pressures are where you want without popping any hoses, move the whole rig into position and then attach the blocks to the cards.

You can still plastic sheet the whole area off and AC that space to keep the humidity down and the ambient air temperature cooler. As for materials, PVC/plastics are easier to work with and won't condense as quickly as metal, nor rust.

That's my $0.02 for what it's worth.
 
sales tax instead of income tax is already very popular in Europe.


but we're 'Merica so we will never be so lucky.... sadly. income tax is a full blown joke.
 
Ac unit is re-sellable with respectable resale value. I agree with that thought. The cost of water cooling will not be recouped when upgrading in the future. A mineral oil setup with a water chiller (like for fish tanks) could work, but seems like more work than its worth.
 
Dude, IF you ever successfully water cool this, I think this will be the most loopy "consumer grade" water loop that's ever looped (which isn't "industrially" designed).

Also, as others have mentioned, you will spend soooo much time installing and hooking up everything if you decide to go water, it'll be just ridiculous. That, and then having to possibly deal with leaks etc.

I think the most "economically viable" alternative if you want to do some kind of liquid-based cooling is to really submerge the whole thing in some kind of liquid and cool that instead... something like this?

http://www.ibtimes.com/secret-bitco...quid-cool-massive-computers-generate-bitcoins

You should get in touch with that guy and ask him about details :-D .
 
I'm most of the way through a degree in heating and air conditioning so I thought I would share my educated observation.

A window air conditioner (or even 2) would not come even close to cooling that room.

15kW (assuming that's the correct value) translates to roughly 51,000 BTU/hour.
In air conditioning terms this would require a 4.5 ton AC unit just to keep up (1 ton of AC is 12,000BTU/h). and probably something more like 5-6 tons to account for summer heat.
4.5 tons is well beyond the power any window air conditioner (usually 0.5 to 1.5 ton), and might even be beyond the range of a ductless mini-split (usually 1-4 tons).

For an idea of what 51,000BTU/hr really means, that amount of energy will heat an entire (well insulated) house to 70F, even when the outside temperature is -20F

Whatever system you eventually chose to cool those cards, you will need to find a way to dispose of that heat outside, or it will literally turn your house into an oven.
 
My advice would be to create a barrier using plastic sheeting around the rack and duct outside air at the bottom of the rack and hot air at the top of the rack directed to the outside.

I design data centers for a living and that's the most cost effective way to cool any IT equipment.

I was going to suggest something similar. Fortunately, this guy setup everything on a wire shelving unit so getting airflow flowing through it won't be a problem.

I would start by getting a couple large AC powered box fans from a local wal-mart and zip tying them to the rack itself to blow air upwards.

If you were going for the bottom to top airflow method then I would get some sort of venting or hood like you would see above a large grill then pump the air outside. Maybe buy or build yourself.


Alternatively, he could place several of the box fans on the front of the rack to blow air front to back, then face the back of the rack towards a window then use a window fan to suck the air out.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Thanks guys for all the ideas, keep them coming.

My original idea before the water cooling was to use a pair of 14" vortex hi performance inline fans and have want draw the air into a room and the other to vent it out. I'd wall off the area where the rigs are so the airflow should cycle the air in the room in under 1 minute (2900 CFM each fan).The ductwork or tubing would be about 25 ft long, so that CFM would drop quite a bit.

http://www.globalindustrial.com/g/hvac/exhaust-fans/in-line-duct-fans/vortex-inline-duct-blower-fans

I'd also like to cut down on the sound considerably as my basement is very large and I have plans to eventually set it up into a workout room along with maybe a bar/entertainment area. If I wall off my systems into a room size of say 15'x10' and maybe used 2 layers of drywall on the inside of the room and 2 layers on the outside, I'd imagine it would cut down on a decent bit of sound. Along with adding some batt insulation inside the wall I'd think it would be relatively quiet to silent as long as the door going into the room was a solid quality fit door. The other idea is to buy Quiet Rock drywall which is suppose to be equivalent to 4-6 sheets of drywall in sound deadening and put that on both the inside and outside on top of the existing drywall..
 
The reason why I like the water cooling setup was the low amount of amps needed to run it. Just cycling the air would be pretty efficient energy wise also. If I decide to air condition the room separately from the main house, I wont even mess with tiny room a/c's. I'll just buy a quality trane 6+ ton a/c unit or other quality name brand and run it. From what I've read, that appears to take around 25 amps at 220v. So that's about an extra $360 a month with electric which isn't to bad. My main panel should still have enough head room so the main isn't running more than 80%. I may also just look into having another 200 amp main lines run if need be. Once I'm fully running, my gpus and other rig that's running will be using 78 amps roughly on a 200 amp panel. So adding an additional A/C to my current unit will tack on roughly 50 amps in the summer so that would put me at 130 amps continuous load minus the rest of the house.
 
If my calculations are correct using a very conservative 290 hashrate of 850 and depending power costs, you're potentially generating $234/day after expenses (not including hardware).

Good lord man.
 
Alright, since a budget wasn't specified..
- purchase a water pump specified for 100% duty cycle, like this one http://www.sears.com/goulds-water-technology-pump-centrifugal-cast-iron-1/p-SPM9211842913
- have discharge line split into 6 lines
- each of the 6 lines will split into 10
- each of the 10 lines will feed a card
- run discharge lines from cards to semi-truck radiator outside
- have cool water discharged into 55 gallon drum
- pump pulls water from 55 gallon drum.

You may want a pressure regulating valve somewhere in there.
 
that is (at current difficulty) about a $75k - $90k (after electric) litecoin miner. This is per year.

very nice.
 
Last edited:
You'd be looking at a couple hundred for each block.
I'd think your plan for this is making money, and while I fully understand the concept of spend money to make money, you don't need to throw it away either. Blocks are poor investments and will bring no return come end-game. Now if you think this setup is a five year plan sort of thing, by all means, but I think that is important information to consider when coming up with a solid solution. ie: If these cards will go up for sale in 6-12 months, it's not worth it.
 
Hell, I should have taken out a loan and made a setup like this. I the mining rig would be more than enough to make the monthly payments...
 
Back
Top