Building a 6 GPU rig for rendering

[jhsu]

So I'm thinking of grabbing 6 GTX 1070 cards for a GPU render rig.

Apparently the requirements for the software are 8x PCIe lanes per card. So I'm looking at a system that could support 6 pcie slots running 8x each and a CPU that supports it.

I've found this motherboard that can support 6 8xpcie lanes:
ASUS X99-E WS (LGA 2011-v3) CEB Intel Motherboard
https://www.newegg.com/Product/Product.aspx?Item=N82E16813182968

And this CPU, which has 40 pcie lanes:
Intel Xeon E5-1620 V4 (LGA 2011-3)
https://www.newegg.com/Product/Product.aspx?Item=N82E16819117696
https://ark.intel.com/products/92991/Intel-Xeon-Processor-E5-1620-v4-10M-Cache-3_50-GHz


I'm actually not very familiar with the LGA 2011 Intel chips, so not sure if that's the best choice compatibility or budget wise.

I think I read in the reviews that I might need to flash the motherboard bios in order to recognize Broadwell chips. I am trying to watch my budget.

 

[SixFootDuo]

Wait, isn't the new Intel HEDT cpu's and motherboard coming out very soon?

 

[jhsu]

I have no idea, but not sure if I want to be spending too much on the processor since it'll mainly be GPU rendering. I might consider up to $500 on the processor depending on the specs. Or do you think prices will come down on the older chips?

 

[IdiotInCharge]

I have no idea, but not sure if I want to be spending too much on the processor since it'll mainly be GPU rendering. I might consider up to $500 on the processor depending on the specs. Or do you think prices will come down on the older chips?

Unless you're interested in going used, price drops aren't likely. New HEDT parts tend to keep their value at retailers (same for Intel's desktop CPUs, for that matter).

Beyond that, I would point out that X299 boards are going to be a bit more flexible on the storage side, and that flexibility might be useful for rendering.

 

[jhsu]

Do you know how the lanes are split up on those 40 lane chips, or where to find that info? I know on my Threadripper it's supposed to have 64 lanes, but after all the other components are taken out there's only 32 pcie graphics lanes.

Yea I don't want to wait, I want to be doing this in the next couple weeks. So I guess you're suggesting a couple generations newer motherboard for an extra $~40? I could only find two x299 boards on Newegg that would support 6 gpus:

The Asus x299 CEB is $510:
https://www.newegg.com/Product/ProductList.aspx?Submit=ENE&IsNodeId=1&N=100007627 601298415 600307108



I checked out the LGA 2066 chips here:
https://en.wikipedia.org/wiki/LGA_2066

I'm definitely not springing for an i9. Running down the charts I think I could settle on a lower middle tier Xeon like the W‑2123 which I could only find on sale direct from Intel on Amazon for $330. The idea of paying more for the motherboard than the cpu seems weird though:
https://www.amazon.com/Intel-BX80673W2123-W-2123-3-6GHz-Fclga2066/dp/B0788BSS85/

Intel server chips confuse the hell out of me. On the wikipedia page I see a W-2125, but there's nowhere in sight that sells them. I guess there's still the question of how the cpu splits up the lanes. Can I assume because the motherboard can support 6 pcie 8x lanes, that a cpu with 48 lanes will support it as well? Or am I completely off base in assuming there's a single system that will support 6 GPU's for rendering?

I can't find anywhere that says how those Xeon chips allocate their pcie lanes.

 

[IdiotInCharge]

I can't find anywhere that says how those Xeon chips allocate their pcie lanes.

This is going to be between the CPU and the motherboard BIOS; I'd recommend digging into the manuls of prospective boards to ensure that your desired configuration- to include storage!- will be properly supported.

 

[jhsu]

Ok, so I found the manual over here:
https://dlcdnets.asus.com/pub/ASUS/...SAGE/Manual/E14511_WS_X299_SAGE_UM_V3_WEB.pdf

In the appendix they have a diagram of the pcie allocations. For the 44 lane block diagram (I'm assuming this is the right chart for the W-2123 even if it's listed as 48 lanes?) it shows the seven slots as 16x 8x 8x 8x 8x 8x 8x, but that already adds up to way more than 44. How is that supposed to work?



Are you saying I can manually allocate the lanes in the bios? I found this spot in the manual:



I think I would be okay with using sata for storage if I needed to dedicate all lanes to the GPUs, I don't know if it works that way.

 

[TheFlayedMan]

Ok, so I found the manual over here:
https://dlcdnets.asus.com/pub/ASUS/...SAGE/Manual/E14511_WS_X299_SAGE_UM_V3_WEB.pdf

In the appendix they have a diagram of the pcie allocations. For the 44 lane block diagram (I'm assuming this is the right chart for the W-2123 even if it's listed as 48 lanes?) it shows the seven slots as 16x 8x 8x 8x 8x 8x 8x, but that already adds up to way more than 44. How is that supposed to work?
View attachment 143382


Are you saying I can manually allocate the lanes in the bios? I found this spot in the manual:
View attachment 143383

The diagram shows 44 lanes coming from the cpu. Theres 36 on the left side and 8 on the right side of the cpu. Its using 2 PLX chips to split the lanes.

 

[Archaea]

I was going to suggest a mining motherboard to do this much cheaper, but they typically only allow 1x lanes to the pci-e slots.

 

[jhsu]

I was going to suggest a mining motherboard to do this much cheaper, but they typically only allow 1x lanes to the pci-e slots.

Yea that's what I thought at first when I started down this rabbit hole, but then realized mining doesn't require a lot of lanes per card, that's why they can gang up so many with splitters on the motherboard.

The diagram shows 44 lanes coming from the cpu. Theres 36 on the left side and 8 on the right side of the cpu. Its using 2 PLX chips to split the lanes.

Yea I just learned about the PLX chip and how it allows the CPU to support more lanes. Right now I'm looking at a WS x299 Sage + i7-9800x which should be able to support 6 pcie slots at 8x.

 

[Deleted member 88227]

If only one of the AIB's would build a thread ripper motherboard with more than 5 x16 slots a TR would be perfect. They have 66 lanes on the CPU.