Server needs better cooling - What options?

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
As the title says I did the (probably big) mistake of buying 2 x Supermicro SC825 TQ-R740LPB 2U chassis. One at the moment is equipped with a X9DR3-F (one socket not working, I will probably need to check for bent pins) and the other with a X9DR3-LN4F+ (both sockets working). Both boards are equipped with a single E5-2620 v1 six core CPU.

It's not that extreme as configuration but the FAN noise is awful. Idling it's still bearable, but as soon as the load increases a slight bit, the noise get's too high for my confort.

I tried to replace the 80mm FANs with quieter ones but then the cooling was not enough and I would throttle because the CPU reached its thermal limit.

For another build (2 x E5-2670 v1) I decided to put it into a Fractal Design Define XL R2 case and am very happy with the results. Cooled by Noctua NH-U12DX i4. Only issues were the standoffs that were not there. Instead of drilling with a dremel because I was afraid of doing lots of damage, I decided to put nylon standoffs in order to just support mechanically the board. Even after that some standoffs were not fitted cause holes of the motherboard aligned with the solid plate of the case ...

I am investigating the following options at the moment:
  1. Put some sound-absorbing foam behind and around the server
    Probably the cheapest althout requires quite some work (and I'm very bad at manual work)

  2. Water cooling with external radiator cooled by silent 120mm fans
    I'm not familiar at all with water cooling. I guess it would be better to buy a closed-loop circuit (some kind of Corsair H80/H90/H110/...) with a Narrow/Square ILM adapter rather than buiding my own loop. The problem with the closed loop is how to put the radiator outside without leaving the case open (this would be easier if I built my own loop but I'm quite afraid of leakages and additionnal maintenance, cleaning the fluid, ...)

  3. Change case
    Build another Fractal Design Define XL R2 configuration and put some e3 xeon-based board at the place of the e5's. Quite expensive solution at 2x130 EUR case + 4x60 EUR coolers + 2x150 EUR PSU

Did any of you have similar issues? What would you reccomend to do?
 
Last edited:

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
I say change case, water cool or both :)
Would you have a water cooling configuration in mind ? There don't seem to be many 2011 Narrow/Square ILM water coolers. I think there are adapters for the Corsair H series on eBay but I'm not sure. The problem would be how to get the "pipe" out since it's a closed loop I need to put the whole water block in through say a PCI slot or something like that. Obviously it's not going to fit :(
 

pendragon1

Fully [H]
Joined
Oct 7, 2000
Messages
22,879
not really that's up to your preferences. id transplant into a new case and custom water cool it all.

this should work for the sq sockets, sq aren't so rare:
http://koolance.com/processor-water-block-cpu-intel-390i

heatkiller has these for narrow:
http://shop.watercool.de/epages/WatercooleK.sf/en_GB/?ObjectPath=/Shops/WatercooleK/Products/18047
ek also has this:
https://www.ekwb.com/shop/ek-annihilator

for aios you can get an asetek gen4 like the corsair h105 and this adapter bracket:
https://store.asetek.com/collection...d-cooler-intel-2011-narrow-retention-ring-kit

you can also always make a hold-down bracket for most normal blocks...

most cases have pass-throughs by the back fan that you can cut notches for the tubes to fit, some case are built like that.

upload_2018-4-24_13-48-58.png


hope this helps.

edited speeling
 
Last edited:
Joined
Apr 5, 2016
Messages
2,205
Watercool, custom loop, external radiators. You certainly don't want to use AIO coolers in a server chassis for... a large number of reasons.

Watercooling is very simple. The number of brands, types of parts, and variety of options can be very overwhelming, but at its core it's a very easy-to-understand concept.

If both your server chassis are in the same rack, you could even save costs by doing a single loop to cool all four sockets. Four CPU blocks, one strong pump, one reservoir, and one big ol' honkin' radiator, probably a 520 or maybe even one of those 9-fan 3x3 "1080" radiators made by Phobya.

Go with simple tubing and barb/hose clamp fittings and it won't hit the wallet too hard.

Your loop would likely look something like this: Reservoir mounted conveniently to the side of your server rack -> pump mounted somewhere below the reservoir -> into the back of the first server -> cpu1 -> cpu2 -> out the back of the first, into the back of the second server -> cpu3 -> cpu4 -> out the back of the second server, into your radiator -> back to your reservoir.

If you do go watercooling, don't be afraid, but spend your time planning, researching, and asking a lot of questions. The legwork will pay off.
 

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
If both your server chassis are in the same rack, you could even save costs by doing a single loop to cool all four sockets. Four CPU blocks, one strong pump, one reservoir, and one big ol' honkin' radiator, probably a 520 or maybe even one of those 9-fan 3x3 "1080" radiators made by Phobya.
Can you provide some more specific references please?

I'm a bit against putting both servers on the same cooling circuit because then the electrical wiring will probably get more complicated, right ? Also thermally the last CPU will have much higher water temperature. Series connection of components also usually increases water pressure.

I assume that both the pump and the fans are powered by one Molex connector or something like that from the PSU.
Otherwise if power supply is external (say 12V) that could work but I don't know how it's feasible to make it more modular. Would still have issues on the thermal side I think though.

About the coolant ... is it really deminaralised water or it's another kind of liquid ?
 
Joined
Apr 5, 2016
Messages
2,205
Can you provide some more specific references please?

I'm a bit against putting both servers on the same cooling circuit because then the electrical wiring will probably get more complicated, right ? Also thermally the last CPU will have much higher water temperature. Series connection of components also usually increases water pressure.

I assume that both the pump and the fans are powered by one Molex connector or something like that from the PSU.
Otherwise if power supply is external (say 12V) that could work but I don't know how it's feasible to make it more modular. Would still have issues on the thermal side I think though.

About the coolant ... is it really deminaralised water or it's another kind of liquid ?
Do you mean references to specific product, or...?

The electrical wiring will be simple: power your fans and power your pump. With a large enough radiator, you won't need the fans to react to the CPU temperatures at all. You'll just set the fans to a speed that you find pleasing noise-wise and let them go; the surface area of the radiator is what bears the brunt of the cooling in this case. I'd personally power it all independently of the servers, perhaps with a standalone controller; if you wanna get fancy.

The last CPU will not have much higher water temperature. This is a common misconception. A component does increase the temperature of the water that flows through it, yes, but that increase in temperature is so insignificant (we're talking a fraction of a degree celcius here) as to have no effect on cooling capacity. There's very little difference in the remaining thermal capacity of 22c water and 27c water; they are both still WAY cooler than a 60c CPU, and will cool it just as effectively.

And yes, the simplest, cheapest and most effective coolant is nothing more than distilled water with a drop or two of growth-inhibitor additive. You can buy much more expensive stuff, but it's not needed. Don't be drawn in by "non-conductive" coolant; it's a lie, it will become conductive over time.

I think you could have a VERY quiet running pair of servers here if you go this route. With a larger rad, you can keep your fans very low - 500 or 600 rpm, say - and still cool like a beast.
 

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
Do you mean references to specific product, or...?
Yes references to specific products would be great, thanks ;)

And yes, the simplest, cheapest and most effective coolant is nothing more than distilled water with a drop or two of growth-inhibitor additive. You can buy much more expensive stuff, but it's not needed. Don't be drawn in by "non-conductive" coolant; it's a lie, it will become conductive over time.
So every say 6 months you need to change your coolant :confused: ?
 
Joined
Apr 5, 2016
Messages
2,205
Yes references to specific products would be great, thanks ;)


So every say 6 months you need to change your coolant :confused: ?
If you just run water and anti-growth, you can run your coolant for much longer. My point was that so-called "non-conductive" coolant is a sham; it's expensive, and it gains conductivity as it leaches ions from the metals in your loop. It's a false sense of security.

I won't be able to go shopping for you until tonight, as I've got to go to work, but I'll happily do it then!
 

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
If you just run water and anti-growth, you can run your coolant for much longer. My point was that so-called "non-conductive" coolant is a sham; it's expensive, and it gains conductivity as it leaches ions from the metals in your loop. It's a false sense of security.
I'm not sure i understand your point about the electrical conductivity.

From what I know the heatspreader of the CPU is supposed to be grounded. I assume so since the coolers I know touch the case chassis, and since this is earthed, it wouldn't be a very good idea that the spreader is at another potential ;). Even if it's not properly grounded the CPU operates at less than 1.3V. So even if your coolant becomes "a bit conductive" I don't think this will cause a big damage. Possibly some EMC issue, noise or unstability, that may be, but not like an explosion or so.

On the other hand, if there is a leak, the whole motherboard will be exposed to (non conductive) water that, after mixing itself with the dust that will be on the motherboard, will become quite conductive. Add to that the higher voltage (12V DC or even 230V AC at the PSU) and you have a problem ...

So I'm not exactly sure how low the conductivity of the coolant needs to be since, in normal operation, you're grounded on both ends (pump, CPU1, CPU2, ...) and, in case of fault, the water will become conductive anyway.

Unless I'm missing something ... :confused:

I won't be able to go shopping for you until tonight, as I've got to go to work, but I'll happily do it then!
Thanks a lot. No need to worry, I think I will have some other xeon e3 builds to do this weekend ;)
 

pendragon1

Fully [H]
Joined
Oct 7, 2000
Messages
22,879
where are you? I just saw your 230v thing...
you can do what VGC is saying with 4 blocks, some tubing, a pump + res, a big rad and some fans. its way simpler than it seems. and everything he said is correct, you dont need fancy coolant or to worry about the cpus being series.

4 Heatkiller blocks ~$235US (2 normal + 2 narrow)
1 watercool mora3 360 + 4x 180mm fans ~$220
1 Alphacool Eisstation pump/res ~$100
12 3/8" barbs + clamps ~$35
3/8" tubing to length ~$10
bioside or antifreeze $5 I use a splash of regular old green antifreeze but you can get the fancy stuff if you want
distilled water at any grocery/drug store $2/g
sata to 6x 3pin fan power cable ~$10
small/cheap/old/used psu to power pump/fans
you can add QDC connections for in-between systems ~$60

https://modmymods.com/watercool-heatkillerr-iv-basic-intel-processor-acetal-18007.html
https://modmymods.com/watercool-hea...l-lga-2011-narrow-ilm-acetal-clean-18047.html
https://modmymods.com/watercool-mo-ra3-360-lt-black-25000.html
https://modmymods.com/phobya-g-silent-180-x-25mm-fan-700rpm-black-79089.html
https://modmymods.com/barrow-g1-4-3-8-barb-fitting-black-tb3c-c02.html
https://modmymods.com/phobya-1-2-13-14-3mm-hexagonal-socket-hose-clamp-black-68128.html
https://modmymods.com/alphacool-eisstation-vpp-reservoir-includes-vpp755-pump-13295.html
https://modmymods.com/modmymods-copper-sulphate-biocide-additive-mod-0014.html
https://modmymods.com/modmymods-3-8-id-x-1-2-od-flexible-pvc-tubing-crystal-clear-mod-0002.html
https://modmymods.com/sata-power-di...-pin-block-fan-hub-power-splitter-cab267.html
 

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
where are you? I just saw your 230v thing...
Sorry for not saying before. I live in Switzerland. I can also order things in Germany though (ship them to a friend :)).

4 Heatkiller blocks ~$235US (2 normal + 2 narrow)
1 watercool mora3 360 + 4x 180mm fans ~$220
1 Alphacool Eisstation pump/res ~$100
12 3/8" barbs + clamps ~$35
3/8" tubing to length ~$10
bioside or antifreeze $5 I use a splash of regular old green antifreeze but you can get the fancy stuff if you want
distilled water at any grocery/drug store $2/g
sata to 6x 3pin fan power cable ~$10
small/cheap/old/used psu to power pump/fans
you can add QDC connections for in-between systems ~$60
Oh my god :eek:. So expensive. That's almost 700$ :confused:. And I'm fairly sure here in Switzerland but also probably in Germany it's going to be more expensive than in the US.
For that price I could almost go with solution c (2 Fractal Design XL cases + 4 x Noctua NH-U12DX i4 + 2 x PSU about 800W ~ 800 EUR). The problem is that this last solution takes quite a lot of space.

About the rack, it's not a "real" server rack. It's like these metallic shelves you put in your garage or something for DYI heavy stuff. Open on all 4 sides, no rails, ...
 

pendragon1

Fully [H]
Joined
Oct 7, 2000
Messages
22,879
Sorry for not saying before. I live in Switzerland. I can also order things in Germany though (ship them to a friend :)).


Oh my god :eek:. So expensive. That's almost 700$ :confused:. And I'm fairly sure here in Switzerland but also probably in Germany it's going to be more expensive than in the US.
For that price I could almost go with solution c (2 Fractal Design XL cases + 4 x Noctua NH-U12DX i4 + 2 x PSU about 800W ~ 800 EUR). The problem is that this last solution takes quite a lot of space.

About the rack, it's not a "real" server rack. It's like these metallic shelves you put in your garage or something for DYI heavy stuff. Open on all 4 sides, no rails, ...
yup either way you go youre looking at a good chunk of change. I found most of that stuff in Europe too for about the same price as most of it is European in origin. heatkiller, alphacool and watercool are all in EU.
 

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
As a side note I should add that these servers will basically be only turned on when needed. Essentially all they will be doing is compiling (DistCC or Icecream clusters).

They generate far too much heat to be able to use them all day long. Also the electricity bill suffers :(

So I would prefer to avoid spending too much extra money into something I probably won't be using a lot.


I'm moving towards smaller servers. I have 3 small E3-1231 v3 servers already running and in the next week I should add 3 more to that list. Much lower power consumption and "modular" (I can turn 2,3,4,5 or 6 on).
 

pendragon1

Fully [H]
Joined
Oct 7, 2000
Messages
22,879
yeah thats a lot of money for limited use. live with the noise and save the cash? its your call whatever you do.
 

Spartacus

2[H]4U
Joined
Apr 29, 2005
Messages
2,064
I know water cooling is popular, but water cooling a server is generally a no-no.

I suppose it depends on your usage, and personal use is a different story.
Personally I don't like the idea of water cooling expensive electronics.
It's not enough benefit for the risk, and you STILL have fan noise on the radiator.

You would never do water cooling on a business server though.

I have a Fractal R5 with stock fans and a Reeven Justice CPU cooler. (8700K @4.8GHz)
Temps are fine and it's right in front of me behind the monitor, I can't hear it at all.

I do hear some fan noise on a Dell server at the desk next to me (not bad though).

ETA: The Fractal R5 has built-in sound deadening material too. Very nice case.

.
 
Joined
Apr 5, 2016
Messages
2,205
I know water cooling is popular, but water cooling a server is generally a no-no.

I suppose it depends on your usage, and personal use is a different story.
Personally I don't like the idea of water cooling expensive electronics.
It's not enough benefit for the risk, and you STILL have fan noise on the radiator.

You would never do water cooling on a business server though.

I have a Fractal R5 with stock fans and a Reeven Justice CPU cooler. (8700K @4.8GHz)
Temps are fine and it's right in front of me behind the monitor, I can't hear it at all.

I do hear some fan noise on a Dell server at the desk next to me (not bad though).

ETA: The Fractal R5 has built-in sound deadening material too. Very nice case.

.
You would absolutely do watercooling on a business server. Heck, fully immersed servers are a real thing, sold that way by vendors.

The risk is pretty minimal if you educate yourself and are careful. The benefit is subjective: you can't put a price on personal preference for noise, for instance, and that's only one area of benefit.

A clutch of 40mm fans at 3000rpm is really not comparable to a 140mm fan at 600rpm, as long as we're talking noise.

Big air coolers are great. They just don't fit in 2u rackmount server chassis. ;)
 

Kardonxt

2[H]4U
Joined
Apr 13, 2009
Messages
3,246
This isn't a fun answer but before you spend thousands on WC components... Is building a proper server room / closet an option? I wouldn't be surprised if a couple insulated walls wasn't a cheaper option.

Real server chassis are loud (especially 2u), heavy, reliable, and offer redundancy \ hotswap capabilities that most residential setups do not. I personally would opt to change the environment to suit real industrial grade equipment rather than needing to custom build \ piece together and maintain multiple special setups every time you add hardware.

The [H] inside me says go all out with fun custom cooling. The business person inside me says keep your equipment standardized, reliable, and hands off.
 
Joined
Apr 5, 2016
Messages
2,205
This isn't a fun answer but before you spend thousands on WC components... Is building a proper server room / closet an option? I wouldn't be surprised if a couple insulated walls wasn't a cheaper option.

Real server chassis are loud (especially 2u), heavy, reliable, and offer redundancy \ hotswap capabilities that most residential setups do not. I personally would opt to change the environment to suit real industrial grade equipment rather than needing to custom build \ piece together and maintain multiple special setups every time you add hardware.

The [H] inside me says go all out with fun custom cooling. The business person inside me says keep your equipment standardized, reliable, and hands off.
You take that business person out back and put the boots to it. We don't need it's kind here! ;)

But f'realzies, Kardon is probably right. There are much cheaper ways to isolate yourself from a noisy machine.
 

Spartacus

2[H]4U
Joined
Apr 29, 2005
Messages
2,064
You would absolutely do watercooling on a business server. Heck, fully immersed servers are a real thing, sold that way by vendors.

The risk is pretty minimal if you educate yourself and are careful. The benefit is subjective: you can't put a price on personal preference for noise, for instance, and that's only one area of benefit.

A clutch of 40mm fans at 3000rpm is really not comparable to a 140mm fan at 600rpm, as long as we're talking noise.

Big air coolers are great. They just don't fit in 2u rackmount server chassis. ;)


>>The risk is pretty minimal if you educate yourself and are careful.

lol... you have no idea who I am or what my credentials are to make such a condescending statement. I'm pretty educated there professor, and there is plenty of risk.
I laugh every time I see somebody lose another mobo or video card to a leak.

I'm an independent IT consultant and have installed hundreds of servers over the years.
In my expert opinion, NO normal small-mid businesses would NOT be the place to install water cooling in a server.

And you're not going to be able to stuff effective water cooling (unless it's external) in a 2U case either.
Such a setup would not make sense in most businesses anyway. There would be absolutely no point.

Yes, I know there is some exotic server equipment being sold.... again, not a thing you'd spec out for a normal small-mid business.


that's not a server...

Golly gee... really?

OP said he was looking at going to a different case/equipment.
The Fractal R5 is a case isn't it?


.
 
Joined
Apr 5, 2016
Messages
2,205
>>The risk is pretty minimal if you educate yourself and are careful.

lol... you have no idea who I am or what my credentials are to make such a condescending statement. I'm pretty educated there professor, and there is plenty of risk.
I laugh every time I see somebody lose another mobo or video card to a leak.

I'm an independent IT consultant and have installed hundreds of servers over the years.
In my expert opinion, NO normal small-mid businesses would NOT be the place to install water cooling in a server.

And you're not going to be able to stuff effective water cooling (unless it's external) in a 2U case either.
Such a setup would not make sense in most businesses anyway. There would be absolutely no point.

Yes, I know there is some exotic server equipment being sold.... again, not a thing you'd spec out for a normal small-mid business.




Golly gee... really?

OP said he was looking at going to a different case/equipment.
The Fractal R5 is a case isn't it?


.
No condescension spoken from me, only perceived by you. I was referring to OP, as they stated plainly that they didn't know much of anything about watercooling.

Can you point out to me where OP stated that these servers were being used in a business? Because yeah, if you're talking ROI, watercooling makes no sense. I believe the commercial use case was an assumption on your part though.

Finally, obviously we're talking about an external loop here. If you'd read any of the thread you'd know that.

Take a chill pill bud. =)
 

pendragon1

Fully [H]
Joined
Oct 7, 2000
Messages
22,879
You would never do water cooling on a business server though.

I have a Fractal R5 with stock fans and a Reeven Justice CPU cooler. (8700K @4.8GHz)
Temps are fine and it's right in front of me behind the monitor, I can't hear it at all.

I do hear some fan noise on a Dell server at the desk next to me (not bad though).
you went business server, normal case I cant hear to dell server. so can you see why I was confusion?
 

Spartacus

2[H]4U
Joined
Apr 29, 2005
Messages
2,064
No condescension spoken from me, only perceived by you. I was referring to OP, as they stated plainly that they didn't know much of anything about watercooling.

Can you point out to me where OP stated that these servers were being used in a business? Because yeah, if you're talking ROI, watercooling makes no sense. I believe the commercial use case was an assumption on your part though.

Finally, obviously we're talking about an external loop here. If you'd read any of the thread you'd know that.

Take a chill pill bud. =)


I read all of the thread, and these two statements were in my responses that you apparently didn't bother reading.


>>I suppose it depends on your usage, and personal use is a different story.

>>And you're not going to be able to stuff effective water cooling (unless it's external) in a 2U case either.


Please try to read and comprehend what people are posting before posting snotty uncalled for responses.
Chill pills don't work against smart-ass know it alls.... but then you know that right?


.
 
Joined
Apr 5, 2016
Messages
2,205
lol... you have no idea who I am or what my credentials are to make such a condescending statement. I'm pretty educated there professor

...and then in your very next post,

Please try to read and comprehend what people are posting before posting snotty uncalled for responses.

Ah, fun fun.

Anyway. OP, if you're working with a garage-grade setup, you could always break out the Dremel and cut a "skylight" in your server cases to make clearance for a big air cooler.
 

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
209
Sorry for not replying sooner, I didn't get any new notification :(.

Anyway. OP, if you're working with a garage-grade setup, you could always break out the Dremel and cut a "skylight" in your server cases to make clearance for a big air cooler.

Unfortunately that's not the case. I'm renting an apartment and no individual garage. Plus I wouldn't put my server in the laundry room because of humidity.
However I noticed that since I moved in this apartment, the walls are much better insulated. I can barely hear the server in the next room (OK at idle fan speed ... but still 3000 rpm or so). Maybe this will not be a problem so big after all ...

At the moment I'm investing some $$$ into second-hand E3 v3 s. Quite cheap, low power and much better efficiency :). Did some benchmarks: 3 x E3-12xx v3 provide about the same performance as a single Dual E5-2670. Except that at idle they pull 3 x 25W or so while the other one pulls like ~ 300 W.
 

BitMaster

Limp Gawd
Joined
Nov 10, 2016
Messages
367
+1 for the MoRA3 and 4 HK's.
I'd even take the bigger MoRa3 and opt for 4x 200mm fans to cool the rad.

That is very very silent too
 
Top