![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
I've been tasked with building a render farm to be put at a colo facility, initially one full rack (47U), render engine being Octane Render, which uses CUDA-capable GPUs in lieu of CPUs. Internally, we went through a number of different render node configs (Intel P4304, Tyan FT77, GTX 690, Titan, 780, 780 Ti) before settling on the following, as having the best price/performance/density ratio:
Motherboard - Asrock Z87 OC Formula
CPU - whatever is cheap at the moment
RAM - 2x 8GB of whatever
GPU - 4x GTX 690, used off ebay/amazon
Case - Chenbro RM41300G
PSU - High Power Rock Solid Pro 1600W
Storage - whatever small SSD is on hand
Cooling - two Delta TFC1212DE and one Delta PFC0812EHE fans.
The finished system looks like this (the cardboard back is due to 41300G not being available at the time this node was built, so we used a regular 41300, which has a 7-slot backplate, so it had to be removed):
Blue is intake, red is exhaust. GTX 690 being a dual GPU card with the fan in the middle, the left front 120mm fan (bottom of the photo, as this is a horizontal rackmount system photographed from above) exhausts the hot air out of the case, while the center and right push cold air in. This works fine in our small server room with two racks, but we're running out of cooling capacity, and there is no way to add more, so we're looking at deploying more boxes at a colo datacenter. However, this presents a problem - whereas in our small server room, barely larger than a closet, exhausting out of both sides is not a problem, a proper datacenter with hot and cold aisles is going to frown upon our rack throwing 5kw+ of heat into the cold row. Therefore, I'm exploring alternative solutions. Right now, I can see five options:
1. Reverse the left 120mm fan to blow in, remove that little duct, and hope for the best.
2. Same as (1), but put a diagonal partition between the left intake fan and the stack of GPUs. This way, the cold air is blown across the top of the GPU stack, where it will be taken in by the fans, while the cold air will go down and to the right, where it will mix with the other intake streams and get blown out the back of the case.
3. Find a place that manufactures custom heatsinks and order a batch of passive, Tesla K10-like coolers to replace the current active ones, then use a TFC1212DE (or two, in series) to blow air through the stack, front to back. The trouble with this approach is that it will most likely require heatpipes to properly distribute heat through the fins, and the bottom plate will need a very complex shape - maybe a custom heatsink manufacturer will be able to use a sample of the stock cooler as a template?
4. Same as (3), but simply remove the fans from the stock heatsinks and rely on external fans to blow air down the length of the cards. The entire area normally occupied by the fan would become wasted space, in comparison to a custom heatsink, but it works well enough with the relatively weak fan in there - a pair of TFC1212DEs will be able to generate a much stronger air stream.
5. Water cooling. Replace stock coolers with water blocks, mount thick 120mm radiators with powerful fans where 120mm fans are currently located, a pair of pumps (in series, for redundancy) in the middle where the 80mm fan currently sits, set the fans to exhaust, so that the hot air is removed from the case rather than blown in, and mount the nodes in reverse, with the faceplate (which now exhausts 1kw+ of heat) towards the hot aisle rather than the cold one. This approach, however, comes with a laundry list of potential problems - for one, I have never even dabbled in liquid cooling, so I lack even a rudimentary experience; for another, it'd be expensive - with GTX 690 water blocks being $130 (four per system = $520), and pumps and radiators also being well in excess of $100 a piece, this would add something like $1k to the cost of each node, pushing it from ~$4k to $5k+ - not excessive, but uncomfortable; and besides that, I'm not at all sure liquid cooled gear would be allowed into a colo facility.
Any other suggestions?
Motherboard - Asrock Z87 OC Formula
CPU - whatever is cheap at the moment
RAM - 2x 8GB of whatever
GPU - 4x GTX 690, used off ebay/amazon
Case - Chenbro RM41300G
PSU - High Power Rock Solid Pro 1600W
Storage - whatever small SSD is on hand
Cooling - two Delta TFC1212DE and one Delta PFC0812EHE fans.
The finished system looks like this (the cardboard back is due to 41300G not being available at the time this node was built, so we used a regular 41300, which has a 7-slot backplate, so it had to be removed):
Blue is intake, red is exhaust. GTX 690 being a dual GPU card with the fan in the middle, the left front 120mm fan (bottom of the photo, as this is a horizontal rackmount system photographed from above) exhausts the hot air out of the case, while the center and right push cold air in. This works fine in our small server room with two racks, but we're running out of cooling capacity, and there is no way to add more, so we're looking at deploying more boxes at a colo datacenter. However, this presents a problem - whereas in our small server room, barely larger than a closet, exhausting out of both sides is not a problem, a proper datacenter with hot and cold aisles is going to frown upon our rack throwing 5kw+ of heat into the cold row. Therefore, I'm exploring alternative solutions. Right now, I can see five options:
1. Reverse the left 120mm fan to blow in, remove that little duct, and hope for the best.
2. Same as (1), but put a diagonal partition between the left intake fan and the stack of GPUs. This way, the cold air is blown across the top of the GPU stack, where it will be taken in by the fans, while the cold air will go down and to the right, where it will mix with the other intake streams and get blown out the back of the case.
3. Find a place that manufactures custom heatsinks and order a batch of passive, Tesla K10-like coolers to replace the current active ones, then use a TFC1212DE (or two, in series) to blow air through the stack, front to back. The trouble with this approach is that it will most likely require heatpipes to properly distribute heat through the fins, and the bottom plate will need a very complex shape - maybe a custom heatsink manufacturer will be able to use a sample of the stock cooler as a template?
4. Same as (3), but simply remove the fans from the stock heatsinks and rely on external fans to blow air down the length of the cards. The entire area normally occupied by the fan would become wasted space, in comparison to a custom heatsink, but it works well enough with the relatively weak fan in there - a pair of TFC1212DEs will be able to generate a much stronger air stream.
5. Water cooling. Replace stock coolers with water blocks, mount thick 120mm radiators with powerful fans where 120mm fans are currently located, a pair of pumps (in series, for redundancy) in the middle where the 80mm fan currently sits, set the fans to exhaust, so that the hot air is removed from the case rather than blown in, and mount the nodes in reverse, with the faceplate (which now exhausts 1kw+ of heat) towards the hot aisle rather than the cold one. This approach, however, comes with a laundry list of potential problems - for one, I have never even dabbled in liquid cooling, so I lack even a rudimentary experience; for another, it'd be expensive - with GTX 690 water blocks being $130 (four per system = $520), and pumps and radiators also being well in excess of $100 a piece, this would add something like $1k to the cost of each node, pushing it from ~$4k to $5k+ - not excessive, but uncomfortable; and besides that, I'm not at all sure liquid cooled gear would be allowed into a colo facility.
Any other suggestions?
