GPU computing build

[TheSpoon]

I was hoping to take advantage of some of your expertise to price out a GPU computing machine. The department I'm in at our university does a lot of computing using CUDA with Nvidia GPUs. We have one ridiculously expensive Tesla machine with 4 GT200-class GPUs. We're looking at buying more GPUs but I'd like to know what you can do with a self-built machine.

Here's the build sticky:
1) What will you be doing with this PC? Gaming? Photoshop? Web browsing? etc
GPU computing
2) What's your budget? Are tax and shipping included?
I'm going to deliberately leave out a budget; I'll just ask that you try to meet the requirements rather than exceed them wildly.
3) Where do you live?
Canada
4) What exact parts do you need for that budget? CPU, RAM, case, etc. Please be very specific.
CPU, GPUs, RAM, PSU, case, motherboard
5) If reusing any parts, what parts will you be reusing? Please be especially specific about the power supply. List make and model.
None
6) Will you be overclocking?
No
7) What size monitor do you have and/or plan to have?
None
8) When do you plan on building/buying the PC?
Now
9) What features do you need in a motherboard? RAID? Firewire? Crossfire or SLI support? etc.
Should have 3 PCI-E x16 slots.
10) Do you already have a legit and reusable/transferable OS key/license? If so, what OS? 32bit or 64bit?
Will use Linux.


Here are some other notes about this build:

• We'd like to maximize the number of GPUs -- I think 3 is about as many as you can have in one desktop-style machine? Note that we're not putting the GPUs in SLI. They'll be used independently.
• The GPUs must be GTX 285s 2GB variants (since Fermi isn't out yet).
• The machine must have at least as many CPU cores as GPU cores. The speed of the CPU is actually mostly irrelevant for us -- the computations are all done entirely on the GPUs.
• The machine must have at least 16GB of RAM, though 24 or 32 would be great.
• I imagine that one of the GPUs will have to be used as a video card when installing the OS? It'd be great if the motherboard had a separate onboard graphics chip, I think. But I'm not sure if such boards exist.

Thanks a lot.

 

[Dangman]

I'm fairly sure that you can find most of these parts in a Canadian store so here's a prelim build:
$460 - 2 x Intel Xeon E5504 CPU
$490 - Supermicro MBD-X8DTH-iF-O Dual LGA 1366 Intel 5520 Extended ATX Motherboard
$548 - 4 x Kingston 4GB DDR3 SDRAM ECC Registered DDR3 1333 RAM
$320 - ENERMAX Galaxy EVO EGX1250EWT 1250W PSU
$340 - Lian Li PC-P80 Case
---
Total: $2158 plus tax and shipping

Both the mobo and case can hold 4 x GTX 280 video cards with ease. There's 16GB of RAM right off the bat but there's still 8 more RAM slots leftover for additional RAM. A total of 8 cores is available just in case you move to dual GPUs or something. That PSU should be more than enough to handle 4 x GTX 280 cards. In addition, that motherboard has onboard video so you don't have to use one of the GTX 280 cards for installing the OS.

 

[TheSpoon]

Here's what I've come up with:

i7 920 - $300
EVGA E760 - $425. The only board I could find with 16x/16x/16x slots.
24GB G.Skill DDR3 - $1350
Antec 1200W PSU - $240
Antec twelve hundred case - $180. I'm particularly unsure about this. Presumably we need a case with good airflow, and this one seems to have a decent heap of fans.
500GB WD Caviar Blue - $50. The machine doesn't need much any disk space locally except that required by the OS.

2GB GTX 285s are actually rather hard to find nowadays, and if we wait long enough we may just get Fermis instead. But either way it'll probably be around $500 per GPU.

That comes to about $4025, depending on exact GPU price.

 

[TheSpoon]

Ooo thanks a lot Danny! The dual CPUs idea gives me something to think about. That motherboard seems to have only x8 PCI-e slots, but this probably isn't a big deal. The onboard video is a big plus.

Does the Lian-Li case provide better cooling than the Antec? And is the Enermax PSU better than the Antec?

I definitely like your build more than mine. That motherboard can hold a ton of memory and I didn't know there were actually boards that could support 4 GPUs, so this is awesome. I'm concerned about cooling the thing though...

Thanks for your recommendations.

 

[Dangman]

Does the Lian-Li case provide better cooling than the Antec? And is the Enermax PSU better than the Antec?

Yes the Lian Li case is better than Antec in terms of cooling and room. And yes the Enermax PSU is of far better quality than the Antec.

 

[sirmonkey1985]

you dont need x16/x16/x16 for computing or cuda.. doesnt draw enough bandwidth since cuda is gpu/shader dependent and mostly shader at that.. you can run a GTX 280 on a 4x slot and still get the same performance out of it being on an x16 slot with cuda applications..


also the limitation of how many gpu's you have depends on the OS you are using.. windows 7 is limited to no more then 8 display outputs.. so that would be 4 cards.. or 2 dual gpu cards and 1 single.. a normal single gpu card counts as 2 display outputs.. a dual gpu card counts as 3 display outputs.. windows xp allows up to 8 gpu's total and no limitation on how many display outputs.. though SLI has a limitation of 3 cards or 4 GPU's..

 

[TheSpoon]

you dont need x16/x16/x16 for computing or cuda.. doesnt draw enough bandwidth since cuda is gpu/shader dependent and mostly shader at that.. you can run a GTX 280 on a 4x slot and still get the same performance out of it being on an x16 slot with cuda applications..

That's been true in my experience. I was looking for a 16x/16x/16x board because it was conceivable to me that some people might have applications that require them to process lots of data and yet perform relatively little computation per datapoint, but that's generally not the case, and unless the data can fit into system RAM, the hard drive would probably end up being the bottleneck in that scenario anyway.

also the limitation of how many gpu's you have depends on the OS you are using.. windows 7 is limited to no more then 8 display outputs.. so that would be 4 cards.. or 2 dual gpu cards and 1 single.. a normal single gpu card counts as 2 display outputs.. a dual gpu card counts as 3 display outputs.. windows xp allows up to 8 gpu's total and no limitation on how many display outputs.. though SLI has a limitation of 3 cards or 4 GPU's..

We'll be using Linux on this machine, and none of the GPUs will be connected to a display. And they won't be in SLI. Should be alright?

 

[TheSpoon]

I just want to make sure that the PSU can really support 2 CPUs and 4 GPUs. According to Nvidia, each 285 consumes up to 204W. Four would consume 816W. That leaves about 434W for the rest of the system. The CPUs have a TDP of 80W. Is it reasonable to suppose they won't draw more than 100W each? Still cutting it pretty close to the power supply's limit.

Thanks.

 

[Dangman]

I just want to make sure that the PSU can really support 2 CPUs and 4 GPUs. According to Nvidia, each 285 consumes up to 204W. Four would consume 816W. That leaves about 434W for the rest of the system. The CPUs have a TDP of 80W. Is it reasonable to suppose they won't draw more than 100W each? Still cutting it pretty close to the power supply's limit.

Thanks.

Nah, 80W is more than likely its max draw. There's still about 272W leftover for the mobo, RAM, and HDD. Should be enough. However if you want to spend the extra cash for peace of mind, go for it:
$400 - SILVERSTONE ST1500 1500W

 

[TheSpoon]

Ah thanks a lot! I was afraid that for four GPUs you'd have to go to some dual-PSU case.

By the way, is there anything terribly wrong with this AMD variant:

$450 - SUPERMICRO MBD-H8DAi+-O Dual 1207(F) AMD SR5690 Extended ATX
$1120 - 8 x 4GB 800MHZ DDR2 ECC REG W/PARITY CL6 DIMM - KVR800D2D4P6/4G
$420 - 2 x AMD OPTERON QUAD-CORE 2378 2.4GHZ SOCKET F 1207

Around here it comes out about $400 cheaper than the Intel.

Thanks.

 

[Dangman]

- Less performance than the Intel
- Uses older and more expensive DDR2 RAM

 

[engiNURD]

Wait for Fermi.

... 2) What's your budget? Are tax and shipping included?
I'm going to deliberately leave out a budget; I'll just ask that you try to meet the requirements rather than exceed them wildly. ...

where's obobski? lol.

 

[TheSpoon]

Wait for Fermi.

Yeah I just wanted to price out a self-built machine. Whether we get it now and get 285s or a few months from now and get Fermis, the price should not change by a large amount.

For reference, the 4-board Tesla machine that we have cost about $7000, and an additional $3500 (or maybe it was $5000, I'm not sure) for a machine to "host" the Tesla machine. Though, each of the Tesla GPUs has 4GB of RAM.