Build for physics calculations

[zakaryah]

I'm looking to build one or more PCs for calculations in a physics lab (mostly large fast Fourier transforms). The task is very computationally demanding, but does not require much RAM (~1GB) or storage (2TB). I have benchmarked GPUs previously and found them to be by far the most cost effective way of doing this efficiently. However, extrapolating from initial tests, I would still require around 12 relatively cutting edge GPUs (nVidia GTX 570 seems ideal in terms of cost to performance). I suspect that it will be cheaper and more efficient to build 3 PCs with 4 cards each rather than 6 with 2, 12 with 1, etc. I would like to build the first machine ASAP. I am looking for suggestions, and comments regarding a rough draft of a machine. I prefer the nVidia cards because I have already written all the code with CUDA.

1) What will you be doing with this PC? Gaming? Photoshop? Web browsing? etc
CUDA programming, optics calculations

2) What's your budget? Are tax and shipping included?
$3,000 for one machine, plus shipping and tax

3) Where do you live?
New York City

4) What exact parts do you need for that budget? CPU, RAM, case, etc. The word "Everything" is not a valid answer. Please list out all the parts you'll need.
CPU, RAM, case, mobo, processor, GPUs, PSU, hard drive, network card

5) If reusing any parts, what parts will you be reusing? Please be especially specific about the power supply. List make and model.
none

6) Will you be overclocking?
no

7) What size monitor do you have and/or plan to have?
irrelevant (can use terminal only!)

8) When do you plan on building/buying the PC?
July - August 2011

9) What features do you need in a motherboard? RAID? Firewire? Crossfire or SLI support? USB 3.0? SATA 6Gb/s? eSATA? Onboard video? etc.
I would like to consider building 1 PC with 4 GPUs (GTX 570). As I understand it, this requires 4 properly spaced PCI-e 2.0 x16 slots, and performance will benefit from a mobo which supports 4 cards at x16 bandwidth. I was considering the ASUS P6T7 but it is very hard for me to determine which mobos ACTUALLY support 4 x16 cards simultaneously. I'm not sure how bad the performance would decline if some cards run at x8, e.g.

10) Do you already have a legit and reusable/transferable OS key/license? If yes, what OS? Is it 32bit or 64bit?
I strongly prefer to use Linux, otherwise I can obtain whatever OS license is best.

Here is my draft; please point out all mistakes I have made in choosing these components. I am especially concerned about how the mobo affects performance:

mobo: ASUS P6T7
processor: Intel i7-960
case: Lian Li Lancool PC-K58
PSU: COOLER MASTER Silent Pro Gold Series RSC00-80GAD3-US 1200W (has 8 6 pin connectors for GPUs)
GPUs: 4 nVidia GTX 570s
RAM, hard drive: not too important
OS: OpenSUSE or Ubuntu, latest versions

 

[Dangman]

Main issue with that choice of mobo is that it will not fit into any normal ATX case. You're gonna have to find a case that speficifically states that it supports CEB mobos if you want to use that Asus mobo.

As for the PCI-E bandwidth issue, at least when it comes to gaming, there's actually very little difference between PCI-E x4 and PCI-E x16. Dunno whether your physics calculations requires more bandwidth or not. If you can, see if you can do your own testing whether or not your applicatiosn really neeed full PCI-E x16 speeds.

Meanwhile I recommend taking a look at this CPU + mobo combo as it has a better price to performance ratio than the setup you have now (assuming that PCI-E bandwidth isn't a major issue):
http://www.newegg.com/Product/ComboDealDetails.aspx?ItemList=Combo.686804