F@H with FPGA accelerator

[Elledan]

First of all, yes I'm aware of the official stance of Stanford on the use of FPGAs for F@H: http://fahwiki.net/index.php/Alternative_FAH_Platforms#FPGA

Second, the reason given on that page doesn't make one bit of sense.

Third, I link to HPRC articles such as the following:
http://www.hpcwire.com/features/FPGA_Floating_Point_Performance.html which shows that double-precision FP math on an FPGA could kick an Opteron's behind even back in 2007.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2440579/ which shows that Molecular Dynamics, which is what F@H (Gromacs) uses for its simulations, runs many times faster on an FPGA than a CPU.

Based purely on these results it doesn't seem unrealistic that FPGAs can outperform GPUs.

An FPGA dev board ranges in price from a few hundred (like the Altera Cyclone II) to many thousands (Virtex-IV) if bought in single units. A separate chip added to a PCIe board or so would be cheaper if done in sufficient volume. This makes it relatively affordable and scales with the demand.

My take and that of my housemate (who happens to be a senior ASIC designer) is that FPGAs are a quite realistic option for F@H and similar HPRC projects and should definitely be looked at.

 

[nitrobass24]

This is pretty interesting.

Would something like this be able to be used with current F@H clients or would a specific client and possibly core have to be written for this.

 

[Elledan]

This is pretty interesting.

Would something like this be able to be used with current F@H clients or would a specific client and possibly core have to be written for this.

There are a few possible approaches. One could use the most generic approach and go basically the PPU route, with the CPU doing the management, sending data to be processed to the FPUs on the FPGA, with the results returned once done. At the other side of the spectrum there is the possibility of implementing the entire core on the FPGA, probably with some external RAM.

Either way some VHDL/Verilog code would have to be written, but if only the parts which would benefit most from the FPGA treatment are ported, the effort required could be kept to a minimum.

As a sidenote, a kind of wrapper could be written maybe as well, which would translate x86/CUDA commands to the native API implemented on the FPGA. Not sure how feasible this would be, however.

 

[keenan]

Faster than a CPU, probably. But faster than a GPU? I really doubt it. GPUs run at high clockrates and are largely just a big pile of FPUs. You might gain some simplicity by moving to an FPGA, but I don't think you can make up for the increased speed and much higher density available on a GPU. And it'd be pretty expensive to try, you could afford to buy 5 high-end video cards for the price of one FPGA dev board.

It'd be interesting to try though, and perhaps a full core implementation could find some major efficiencies that can't translate to a CUDA implementation.

 

[Elledan]

Faster than a CPU, probably. But faster than a GPU? I really doubt it. GPUs run at high clockrates and are largely just a big pile of FPUs. You might gain some simplicity by moving to an FPGA, but I don't think you can make up for the increased speed and much higher density available on a GPU. And it'd be pretty expensive to try, you could afford to buy 5 high-end video cards for the price of one FPGA dev board.

It'd be interesting to try though, and perhaps a full core implementation could find some major efficiencies that can't translate to a CUDA implementation.

Actually GPU FPUs run at lower clockspeeds than the average CPU FPU. They're also heavily specialized in graphics-related operations, which makes them far less efficient at things like MD algorithms. Especially things like memory transfers are glacially slow with GPUs. Transferring anything between global and local memory/registers takes many, many cycles. Transferring between system RAM and VRAM takes about two ice ages.

Since in 2007 an MD algorithm could run 5-10x faster on a 2004-era FPGA than a respectable CPU from that time, and FPGAs keep adding more and faster logic elements, I'm quite sure an FPGA would be on-par or beat the most powerful GPU you can currently or soon buy.

Also, in volume FPGAs really aren't that expensive. Yes, dev boards are expensive, but that's because they include a lot of stuff a production board would never include, like full debugging support and more I/O than you can shake a stick at. There's a reason why FPGAs are used quite often in low-volume electronics products, including high-end soundcards.

 

[keenan]

Actually GPU FPUs run at lower clockspeeds than the average CPU FPU.

Well yeah, but faster than an FPGA, which is what we're talking about. Looks like Virtex 6 max clock speed is ~600MHz, and that's if you can route your synthesis to actually run stable at anything close to that speed, which is pretty darn unlikely, you'd be very lucky to get half that.

Since in 2007 an MD algorithm could run 5-10x faster on a 2004-era FPGA than a respectable CPU from that time, and FPGAs keep adding more and faster logic elements, I'm quite sure an FPGA would be on-par or beat the most powerful GPU you can currently or soon buy.

The MD paper seems to come to the conclusion that a modern (and quite expensive, I might add) Virtex 5 would be about 10-17x faster than software on their test system (which isn't well described, but 2.8GHz Xeon and research likely done in 2007 leads me to believe it's probably a NetBurst core unit). That's significant, but considering modern CPUs have double the cores and much, much better FPU performance I'd expect they bring that gap down 4-5x to 2-3x faster. Current GPU cores for F@H (not that they're directly comparable, but...) seem to be doing about the same or slightly better.

The paper points out that one of the main performance issues is implementing the FPUs needed with FPGA logic instead of using hard blocks. I wonder if it would be useful to deeply pipeline the process, implement everything other than FPU on the FPGA and then offload FPU operations to the graphics card. If you could get enough bandwidth...

Also, in volume FPGAs really aren't that expensive. Yes, dev boards are expensive, but that's because they include a lot of stuff a production board would never include, like full debugging support and more I/O than you can shake a stick at. There's a reason why FPGAs are used quite often in low-volume electronics products, including high-end soundcards.

The range of FPGA/CPLD products available is pretty darn wide. I'm looking at a small CPLD for a hobby project I'm working on (coincidentally it's audio related ) that's about $1.50/unit in small quantities. FPGAs start at about $10/unit in singles. But the ones in question here are in the $100s - even ordering a relatively large quantity isn't going to get you into double digits, and that's not getting started on what it costs for a small-run of suitable PCBs, or having them assembled.

I'd be pretty curious to see what could be done, FPGAs do offer a metric buttload of processing power if you can make your core simple enough, I'm just doubting that any FPU-heavy application that can be efficiently pipelined is going to beat a GPU on one.

 

[Elledan]

I'd be pretty curious to see what could be done, FPGAs do offer a metric buttload of processing power if you can make your core simple enough, I'm just doubting that any FPU-heavy application that can be efficiently pipelined is going to beat a GPU on one.

For me it's primarily a matter of curiousity as well Those articles I linked show that FPGAs at least aren't hopelessly behind CPUs when it comes to FP performance, and since virtually every single transistor which goes into a new FPGA design is usable for logic blocks while clockspeed keeps increasing, whereas for CPUs FPU-wise all one can do is fiddle with the clockspeed (two vs one axes of improvement), I'd very much like to see an update on this research to see whether FPGAs are now in relation to CPUs and GPUs.

If I had an FPGA dev kit I'd do it myself, but alas

 

[Vaulter98c]

Times like these I wish I didn't change my major and stuck with computer programming / video game design

damn you Cisco ENT/CNA
Posted via [H] Mobile Device

 

[Elledan]

Fight the system, never specialize