4-GPU Cuda Super PC

[leSLIe]

A physicist friend of mine needs a new PC for some heavy GPU simulations, it would be connected and used as a small cluster through the network, this is what he needs:

1) What will you be doing with this PC? Gaming? Photoshop? Web browsing? etc
CUDA Programming, GPU simulations
2) What's your budget? Are tax and shipping included?
US$6000 before taxes and shipping
3) Where do you live?
Austria (but it will be bought in the US)
4) What exact parts do you need for that budget? CPU, RAM, case, etc. The word "Everything" is not a valid answer. Please list out all the parts you'll need.
CPU, Mobo, RAM, 4 GPUs, HDs (not SSD), DVD-burner,PSU, case, network card, linux friendly raid card
5) If reusing any parts, what parts will you be reusing? Please be especially specific about the power supply. List make and model.
monitor
6) Will you be overclocking?
no
7) What size monitor do you have and/or plan to have?
18"
8) When do you plan on building/buying the PC?
ASAP
9) What features do you need in a motherboard? RAID? Firewire? Crossfire or SLI support? USB 3.0? SATA 6Gb/s?etc.
At least Corei7; RAID; GPUs with lots of RAM; special editions are welcome; at the very least 16GB of RAM; very fast network card; all the four PCIe slots are to be 16X (if possible); Maybe 2 CPUs
10) Do you already have a legit and reusable/transferable OS key/license? If so, what OS? Is it 32bit or 64bit?
Linux will be used

 

[ultimatedesk]

Wow, that's a really interesting request.

Not sure where I would start in terms of sourcing hardware, since most of my experience with system building is with "residentially" available hardware. What vendors sell those cool Tesla GPU's?!

Anyways, I'm not sure if you have much experience building these types of computers, but I found a few links that might be handy for you:

About GPU Processing with nVidia

Sample Workstations (Ultra High-End Stuff)

Super Expensive Tesla nVidia Card

This project sounds like it would be a lot of fun. Not sure what you can get for 6 large ones in this area of computing. So let us know what you find and what you end up selecting!

 

[leSLIe]

Wow, that's a really interesting request.

Not sure where I would start in terms of sourcing hardware, since most of my experience with system building is with "residentially" available hardware. What vendors sell those cool Tesla GPU's?!

Anyways, I'm not sure if you have much experience building these types of computers, but I found a few links that might be handy for you:

About GPU Processing with nVidia

Sample Workstations (Ultra High-End Stuff)

Super Expensive Tesla nVidia Card

This project sounds like it would be a lot of fun. Not sure what you can get for 6 large ones in this area of computing. So let us know what you find and what you end up selecting!

I'll look into those links
And yes, they already have a Tesla cluster (although not the latest 6GB Tesla), they just want something "smaller" and cheaper, and yes the extra on-board GPU RAM is a must!!

 

[Dangman]

$300 - Intel Core i7-2600 CPU
$270 - Asus P8P67 WS Revolution Intel P67 Motherboard
$156 - 2 x G.Skill Value Series F3-10600CL9D-8GBNT 2 x 4GB DDR3 1333 RAM
$1120 - 4 x Palit NE5X56T01142-1041F GeForce GTX 560 Ti 2GB PCI-E Video Card
$300 - Antec High Current Pro HCP-1200 Modular 1200W PSU
$80 - Lian Li Lancool PC-K58 ATX Case
$25 - Cooler Master STB-3T4-E3-GP 4 in 3 Device Module w/ 120mm Fan
----
Total: $2251

Some notes:
- That motherboard has a built-in Intel NIC so no need to buy a separate Intel NIC.
- I added the Coolermaster Module so that you can fit in a total of 8 drives into that case.
- That case has 8 expansion slots in the rear which allows you to fit in four dual-slot video cards just fine.
- If you use Linux software RAID, you can use all 8 ports as part of one RAID array

 

[plugwash]

I'm not convinced LGA1155 is the best choice for this application. With four cards active at the same time you will only have an average of four lanes per card (x8 from each card to the bridge but only x16 total from the bridge to the CPU).

With x58 you can get all cards on a true x8 to the IO hub and with dual 5520 you can get all cards on x16 to the IO hubs (the only board I know of that offers this configuration is the X8DTG-QF).

 

[leSLIe]

I'm not convinced LGA1155 is the best choice for this application. With four cards active at the same time you will only have an average of four lanes per card (x8 from each card to the bridge but only x16 total from the bridge to the CPU).

very interesting

With x58 you can get all cards on a true x8 to the IO hub and with dual 5520 you can get all cards on x16 to the IO hubs (the only board I know of that offers this configuration is the X8DTG-QF).

you mean the Supermicro dual cpu mobo?

 

[plugwash]

(the only board I know of that offers this configuration is the X8DTG-QF).

you mean the Supermicro dual cpu mobo?

Thats the one, dual CPU and dual IO hub.

http://www.supermicro.com/products/motherboard/QPI/5500/X8DTG-QF.cfm

 

[Dangman]

I'm not convinced LGA1155 is the best choice for this application. With four cards active at the same time you will only have an average of four lanes per card (x8 from each card to the bridge but only x16 total from the bridge to the CPU).

Well the motherboard I chose actually has the Nvidia NF200 chip to provide additional PCI-E lanes. So each of the four PCI-E lanes will have at least x8 speeds. So while not full x16 for all four slots, x8 is still decent enough

 

[plugwash]

Well the motherboard I chose actually has the Nvidia NF200 chip to provide additional PCI-E lanes. So each of the four PCI-E lanes will have at least x8 speeds.

The NF200 is a PCIe bridge. It routes traffic to devices and IIRC it has some tricks on top of that too but it isn't magic. If your system is trying to communicate with all cards at the same time then the bottleneck will become the x16 link from the NF200 to the CPU.

Personally i'd say x58 is probablly the best compromise, I don't think the extreme cost of going dual processor with dual IO hub (the trouble is the only board I know of that offers that configuration is in a propietry form factor and the corresponding case is EXTREMELY expensive) is likely to be worth it (though it would be within the budget I think).

 

[pxc]

That's a high budget. How abouts this?

$329.99 ASUS P6T6 WS Revolution with NF200 3xPCIe true x16 LGA 1366 Intel X58 ATX Intel Motherboard 6 PCI-E x16, Support CrossFireX/SLI, SASsaby Card
$999 i7 990x 6 core 3.46GHz (coming in a few weeks)
$299.99 G.SKILL Ripjaws Series 24GB (6 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model F3-10666CL9T2-24GBRL
4 x $349.99 EVGA 012-P3-1570-AR GeForce GTX 570 (Fermi) 1280MB 320-bit GDDR5 PCI Express 2.0 x16 HDCP Ready SLI Support Video Card

and I'll just steal the rest from above:

$300 - Antec High Current Pro HCP-1200 Modular 1200W PSU
$80 - Lian Li Lancool PC-K58 ATX Case
$25 - Cooler Master STB-3T4-E3-GP 4 in 3 Device Module w/ 120mm Fan

total: $3415

I chose the GTX 570 because of better reference cooling. It uses the in through front side of case, out through back slot. It won't turn the inside of the case into a furnace like the GTX 560 Ti reference cooler. The GTX 560 Ti has better performance/$ though.

This is still way under max budget. I want to recommend a DP workstation board, but I didn't see any with 4 PCIe x16 slots that I liked.

edit: scratch that board. The top slot isn't where it needs to be to support 4 cards. Looking...
$380 EVGA X58 Classified 4-Way SLI Motherboard - LGA 1366, Intel X58, SATA, 4-Way SLI Ready, Ready, Triple Channel DDR3, RAID, Hyperthreading (newegg is OOS right now)

 

[plugwash]

That's a high budget. How abouts this?

$329.99 ASUS P6T6 WS Revolution with NF200 3xPCIe true x16

More half truths from motherboard manufacturers I see, I wouldn't count using a bridge chip as 3x true x16.

 

[pxc]

More half truths from motherboard manufacturers I see, I wouldn't count using a bridge chip as 3x true x16.

It's really not a bottleneck. Not sure why you make it such an issue.

Communication with main memory is going to be the bottleneck, if there is one.

 

[Dangman]

The NF200 is a PCIe bridge. It routes traffic to devices and IIRC it has some tricks on top of that too but it isn't magic. If your system is trying to communicate with all cards at the same time then the bottleneck will become the x16 link from the NF200 to the CPU.

Hmmmm, still dunno if that's gonna be as big of an issue though.

Personally i'd say x58 is probablly the best compromise, I don't think the extreme cost of going dual processor with dual IO hub (the trouble is the only board I know of that offers that configuration is in a propietry form factor and the corresponding case is EXTREMELY expensive) is likely to be worth it (though it would be within the budget I think).

Well here's a X58 based setup anyway:
$300 - Intel Core i7 950 CPU
$365 - Asus Rampage III Extreme LGA 1366 Intel X58 ATX Motherboard
$231 - 3 x G.Skill Value Series F3-10600CL9D-8GBNT 2 x 4GB DDR3 1333 RAM
---
Total: $896 plus tax and shipping.

Same GPU, PSU, case, and 4in3 Device module as before. The thing that bugs me about a X58 setup is the price of the Core i7 950: Just as much as the much faster Core i7-2600K CPU.

 

[plugwash]

For comparison here is how the supmicro dual CPU/dual IOH soloution would work out (i'm not going to list GPUs because I don't follow what is good and bad in them)

Supermicro X8DTG-QF motherboard arround $500
Supermicro 747TQ-R1400B case with 1.4KW redundant PSUs arround $1000 (I think this is a ripoff but the board is a propietry form factor so you don't really have much choice)
2x Supermicro SNK-P0040AP4 heatsinks arround $30 each
2x xeon 5620 arround $400 each
Ram 6x4GB ecc arround $360

So you are talking $2700 or so before you start adding GPUs and drives. OTOH for that you get full x16 to all your GPUs, a lot of CPU power for those tasks that can't be done on GPUs and you can accomodate your GPUs while still leaving slots free for other stuff. Personally I don't think it's worth it but I thought i'd work out the costing on it anyway.

 

[pxc]

^^ not bad within the budget, at least if it's a good board. A pair of Westmere quad core 2.53GHz Xeons (32nm, 80W) is a little over $1000 for both.

 

[TheInfernal]

The performance advantage in a dual CPU setup due to having two extra cores over the 980X is easily offset with the extra clock speed and overclockability of the 980X.
A $300 4-core Sandy Bridge 2600K matches the $1000 6-core Nehalem 980X and overclocks better.

Therefore, you should save your money and buy the 2600K instead of a dual CPU setup or the 980X.
Edit: Unless 4X is a bottleneck*

 

[pxc]

I'd rather have a workstation class board than a consumer board, 1s or 2s. TBH, I'd rather buy a major OEM workstation in this situation, but the OP asked for a build so there it is.

 

[tonytnnt]

I was able to get a Colfax CXT5000 PSC (http://www.colfax-intl.com/ms_tesla.asp?M=102) configured for $4406 with 2 quad core Xeon X5550s and 16 gigs of RAM (8 x 2GB registered ECC). You could then populate the slots with four graphics cards and be under budget. (for the config, select 2 X5550s, 8 x 2 GB RAM, 250 GB HDD, and Lite On DVD Drive under BD/DVD-ROM). You could go even cheaper by going with one quad core or even a dual core proc. This is assuming they'd sell it to you without a Tesla of course...

Edit:

You can configure a 12 core AMD system with 16 GBs of RAM and 4 5870s for under $4500 if your physicist friend is open to/interested in ATI Stream.

I'll work on my own build it yourself ideas in a bit. I've been hoping to experiment with some groundwater flow models on GPUs for awhile now...

 

[pxc]

If you need lots of memory on the GPUs, the two options right now are: 4GB on the $1300 (each) Tesla C1060 (similar to GTX 280, but more memory) or 3GB on the $550 each Palit GTX 580, which will be a cooling nightmare in a desktop board due to placement of the fans.

The faster Tesla models are not within your budget. The reasonably priced FireStream cards top out at 2GB. Don't know if CUDA is a requirement on the build.

 

[mutantmagnet]

For once that I've been here the Hardforum template is inadequate.

You're going to have to talk more with your friend and get some additional details if you don't have them already and have a better idea of the differences between the different tiers of video cards.

Saving him $3000 on gpus is pointless if it he has to deal with down time for various reasons. Also Nvidia likes to gimp their products in specific ways to make the pricing of their products appropriate for the targeted markets. If his work involves doing stuff beyond the physical limits of GEforce they become a pointless choice.

Still you also need to consult other people on the matter if you haven't done so already.

 

[leSLIe]

For once that I've been here the Hardforum template is inadequate.

You're going to have to talk more with your friend and get some additional details if you don't have them already and have a better idea of the differences between the different tiers of video cards.

Saving him $3000 on gpus is pointless if it he has to deal with down time for various reasons. Also Nvidia likes to gimp their products in specific ways to make the pricing of their products appropriate for the targeted markets. If his work involves doing stuff beyond the physical limits of GEforce they become a pointless choice.

Still you also need to consult other people on the matter if you haven't done so already.

Well yes, and no
They will buy a new Tesla cluster, but not right now.

They work with CUDA, so AMD cards are out of the picture
Right now, they need a small GPU cluster, this one, for some "minor" simulations, so GeForces would do fine, GTX580 would do. And a good setup to be able to squeeze 16X out of those four PCIe slots

 

[pxc]

Also Nvidia likes to gimp their products in specific ways to make the pricing of their products appropriate for the targeted markets.

That's a good point, sort of. The DP compute capability varies with different models. The GTX 460, for example, only had 2/3 the DP capability per SP (on average) as the GTX 470/480 (460 = DP is 1/12th of SP throughput, 470/480 = DP is 1/8th of SP throughput). The GTX 560 Ti though has full DP throughput compared to the GTX 580, with fewer SPs of course. The Tesla models have 4 times the peak DP throughput as the consumer models. That last sentence may be the most important for the OP's requirements.

Even though the GTX 460 was "gimp(ed)", it was quite a bit better than what AMD did to some models by completely removing DP capability altogether. http://www.geeks3d.com/20091014/radeon-hd-5770-has-no-double-precision-floating-point-support/ And you can see that even when gimped, the GeForce models pretty much dominate AMD's current offerings in GPGPU: http://www.anandtech.com/show/4135/nvidias-geforce-gtx-560-ti-upsetting-the-250-market/15

Selective outrage is fun.

 

[Blue Fox]

This is probably the best barebone and starting point you can get for what you're doing: http://www.newegg.com/Product/Product.aspx?Item=N82E16816101282

 

[plugwash]

that barebone is the same combination I suggested just with the motherboard already in the case.

 

[aviat72]

If you are going towards a Xeon based solution, then keep in mind that the pricing of these chips follows an exponential curve. You can get an E5620 at $400, while the E5640 is about $800. The difference in performance is not very much but Intel is the master of market segmentation and value based pricing.

Also if you go the Xeon way make sure the chipset is 5520 and it has enough PCI-Ex lanes. The 5500 has a lower number of lanes.

 

[leSLIe]

If you are going towards a Xeon based solution, then keep in mind that the pricing of these chips follows an exponential curve. You can get an E5620 at $400, while the E5640 is about $800. The difference in performance is not very much but Intel is the master of market segmentation and value based pricing.

Also if you go the Xeon way make sure the chipset is 5520 and it has enough PCI-Ex lanes. The 5500 has a lower number of lanes.

well I'm sure they can go with those $400 Xeons

 

[plugwash]

Also if you go the Xeon way make sure the chipset is 5520 and it has enough PCI-Ex lanes. The 5500 has a lower number of lanes.

To clarify there are three chipset combinations i've seen for LGA1366 dual socket xeons.

5500 IOH + ICH10 (less PCIe than x58)
5520 IOH + ICH10 (same PCIe as x58)
Dual 5520 IOH + ICH10 (more PCIe than x58)

The supermicro board I mentioned has the third of these (hence why it can offer four true x16 slots).

 

[mq6600]

Motherboard- EVGA Classified SR-2 270-WS-W555-A2 $589.99
Processor(s)- 2x Intel Xeon E5620 Westmere 2.4GHz $779.98
Memory- 2x CORSAIR DOMINATOR 24GB (6 x 4GB) 1333 $799.98
Power Supply- SILVERSTONE ST1500 1500W ATX $379.99
GPU- 4x EVGA GeForce GTX 580 $1,999.96
Case- LIAN LI PC-V2120B Black Aluminum $429.99
HDD- Western Digital Caviar Black WD1501FASS 1.5TB $119.99
DVD Burner- LITE-ON CD/DVD Burner $17.99

Grand total $5,117.87
and you still have enough for a ladies night!

All this can be found on the Egg

 

[mq6600]

To clarify there are three chipset combinations i've seen for LGA1366 dual socket xeons.

5500 IOH + ICH10 (less PCIe than x58)
5520 IOH + ICH10 (same PCIe as x58)
Dual 5520 IOH + ICH10 (more PCIe than x58)

The supermicro board I mentioned has the third of these (hence why it can offer four true x16 slots).

I think the SR-2 has the 5520 IOH + ICH10 and a nf200 bridge. Correct me if im wrong.

 

[leSLIe]

Motherboard- EVGA Classified SR-2 270-WS-W555-A2 $589.99
Processor(s)- 2x Intel Xeon E5620 Westmere 2.4GHz $779.98
Memory- 2x CORSAIR DOMINATOR 24GB (6 x 4GB) 1333 $799.98
Power Supply- SILVERSTONE ST1500 1500W ATX $379.99
GPU- 4x EVGA GeForce GTX 580 $1,999.96
Case- LIAN LI PC-V2120B Black Aluminum $429.99
HDD- Western Digital Caviar Black WD1501FASS 1.5TB $119.99
DVD Burner- LITE-ON CD/DVD Burner $17.99

Grand total $5,117.87
and you still have enough for a ladies night!

All this can be found on the Egg

looking good

 

[plugwash]

For this purpose the MB/PSU/case that MQ6600 suggested seems inferior to the supermicro barebones mentioned above ( http://www.newegg.com/Product/Product.aspx?Item=N82E16816101282 ) and very nearly as expensive. It is inferior in the following ways

1: with the supermicro even with 4 dual slot GPUs in you still have three slots free for other stuff, with the EVGA with four dual slot GPUs in every slot is either occupied or blocked.
2: with the supermicro all the GPUs get a direct x16 connection to an IOH, with the EVGA they are behind NF200 bridges

I'd also seriously consider replacing that desktop ram with ECC server ram.

Edit: typo in first sentance, wrote CPU instead of PSU

 

[pxc]

lol, there are so many frustrating things about pricing out a top end system.

Build one soon, and in several months:
Kepler-based GPUs come out with twice the compute capability
Patsburg 2s boards come out at the same price as a 2s last or prior gen workstation board costs now

Maybe Danny Bui's original $2250 configuration isn't too bad to wait out a larger upgrade later, or at least with one of the revised boards containing more x16 slot bandwidth.