After poking around looking for solutions to a couple of problems I've noticed with my Multi GPU setup, I've decided to start a discussion about one of the issues I found.
It just might be that the [H]orde is leaving many hundreds if not thousands of PPD on the table due to this issue and our lack of knowledge about it. That's just unacceptable!
So let's see how many people it may potentially be affecting.
If you run 2 or more GPUs on the same board and they are different types (9600GT/8800GT, 8800GS/9800GTX, etc) you may get a lower PPD value on the second card compared to what it would get running alone or paired with a card of the same type. Those of you that ran SLI (I'm not sure this affects Crossfire or ATI cards in general. I don't think it does but am not sure.) and have matched cards or those that just have matched card types (9600GT/9600GT OC, 8800GT/8800GT, etc) shouldn't be affected.
I first found this out (other than I noticed it myself based on what my 8800GS should have been putting out) looking around at the FF in this thread.
There's much speculation as to whether it may be a chipset issue, driver issue or other hardware limitation/issue (such as the GPU).
and from the Cuda Documentation
So if anyone is running a mixed (or matched) set, please post up and lets see if we are seeing the same issues. If you need a good reference to what your card should be getting PPD wise here is a direct RS download link to a great little spreadsheet that legoman666 from AT is doing. It gives average speeds for most cards as well as OC settings for anyone unsure of how high your card may be able to clock. So far this link always gets his latest updated version.
Bottom line is if you're running dual cards or want to run dual cards, keep them matched up. They don't have to be running at the same speed OR have the same amount of ram (8800GT 256/8800GT 512 is OK) but they do need to be the same type. If there's free points on the table lets pick um up!
It just might be that the [H]orde is leaving many hundreds if not thousands of PPD on the table due to this issue and our lack of knowledge about it. That's just unacceptable!
So let's see how many people it may potentially be affecting.
If you run 2 or more GPUs on the same board and they are different types (9600GT/8800GT, 8800GS/9800GTX, etc) you may get a lower PPD value on the second card compared to what it would get running alone or paired with a card of the same type. Those of you that ran SLI (I'm not sure this affects Crossfire or ATI cards in general. I don't think it does but am not sure.) and have matched cards or those that just have matched card types (9600GT/9600GT OC, 8800GT/8800GT, etc) shouldn't be affected.
I first found this out (other than I noticed it myself based on what my 8800GS should have been putting out) looking around at the FF in this thread.
I noticed a few users running multiple GPU's in one machine having similar ppd across all of the cards in the system, regardless if it's 16x or 4x PCI-E. This was somewhat perplexing as I have two machines that I had been trying to acheieve similar results with but couldn't figure out why I was not able to. So I've just finally figured it out, as long as you are using the same type of cards in the system, output will be nearly equal at a given clock speed. However, if you mix cards, say, an 8800 GT with a 9600 GT..... or even a 9800 GTX with a 9600 GT, you will get a significant drop in performance from the second and third cards in the system while the primary card will output as normal. In the above situation, the 9800GTX put out a solid 5300ppd on its own, while the clocks on my 9600 GT have it putting out around 4100ppd. Together, the 9800GTX maintains its 5300ppd but the 9600GT drops to only 3400ppd.
I confirmed this a few minutes ago by swapping out the 9800 GTX with another 9600 GT, giving me two in the system. Both 9600 GT's are now putting out almost identical ppd near 4100.
There's much speculation as to whether it may be a chipset issue, driver issue or other hardware limitation/issue (such as the GPU).
I went searching through CUDA documentation last night and really didn't come up with anything. Although in the SDK there is the MonteCarlo.exe MonteCarloMultiGPU.exe. Interesting that the multiGPU program has a ratio of 1.58 if I divide the iterations of the faster card by the iterations of the slower card, which is the same ratio I get when dividing the PPD of the faster card with the slower. Don't know what use that info is to you guys. Guess it proves it is not a FAH issue(which I think we already knew), and it also provides a faster way to check after you make changes(runs in like 10 secs) vs running at minimum 1 FAH frame which for the slower card is around 2+ minutes. There was some other talk in various CUDA forums about heterogeneous cards in the same system but no one had any definate answer.
and from the Cuda Documentation
Page 16, section 3.2 "The use of multiple GPUs as CUDA devices by an application running on a multi-GPU system is only guaranteed to work if theses GPUs are of the same type." The question really is what does this exactly mean for our application: does that mean it may run slower, not at all, and more importantly any solution to get it to work at full speed. It also says "an application" we are actualling using 2 applications" very cloudy indeed
So if anyone is running a mixed (or matched) set, please post up and lets see if we are seeing the same issues. If you need a good reference to what your card should be getting PPD wise here is a direct RS download link to a great little spreadsheet that legoman666 from AT is doing. It gives average speeds for most cards as well as OC settings for anyone unsure of how high your card may be able to clock. So far this link always gets his latest updated version.
Bottom line is if you're running dual cards or want to run dual cards, keep them matched up. They don't have to be running at the same speed OR have the same amount of ram (8800GT 256/8800GT 512 is OK) but they do need to be the same type. If there's free points on the table lets pick um up!