Mix & Match your GPUs

[nomad8u]

After poking around looking for solutions to a couple of problems I've noticed with my Multi GPU setup, I've decided to start a discussion about one of the issues I found.

It just might be that the [H]orde is leaving many hundreds if not thousands of PPD on the table due to this issue and our lack of knowledge about it. That's just unacceptable!
So let's see how many people it may potentially be affecting.

If you run 2 or more GPUs on the same board and they are different types (9600GT/8800GT, 8800GS/9800GTX, etc) you may get a lower PPD value on the second card compared to what it would get running alone or paired with a card of the same type. Those of you that ran SLI (I'm not sure this affects Crossfire or ATI cards in general. I don't think it does but am not sure.) and have matched cards or those that just have matched card types (9600GT/9600GT OC, 8800GT/8800GT, etc) shouldn't be affected.

I first found this out (other than I noticed it myself based on what my 8800GS should have been putting out) looking around at the FF in this thread.

I noticed a few users running multiple GPU's in one machine having similar ppd across all of the cards in the system, regardless if it's 16x or 4x PCI-E. This was somewhat perplexing as I have two machines that I had been trying to acheieve similar results with but couldn't figure out why I was not able to. So I've just finally figured it out, as long as you are using the same type of cards in the system, output will be nearly equal at a given clock speed. However, if you mix cards, say, an 8800 GT with a 9600 GT..... or even a 9800 GTX with a 9600 GT, you will get a significant drop in performance from the second and third cards in the system while the primary card will output as normal. In the above situation, the 9800GTX put out a solid 5300ppd on its own, while the clocks on my 9600 GT have it putting out around 4100ppd. Together, the 9800GTX maintains its 5300ppd but the 9600GT drops to only 3400ppd.
I confirmed this a few minutes ago by swapping out the 9800 GTX with another 9600 GT, giving me two in the system. Both 9600 GT's are now putting out almost identical ppd near 4100.

There's much speculation as to whether it may be a chipset issue, driver issue or other hardware limitation/issue (such as the GPU).

I went searching through CUDA documentation last night and really didn't come up with anything. Although in the SDK there is the MonteCarlo.exe MonteCarloMultiGPU.exe. Interesting that the multiGPU program has a ratio of 1.58 if I divide the iterations of the faster card by the iterations of the slower card, which is the same ratio I get when dividing the PPD of the faster card with the slower. Don't know what use that info is to you guys. Guess it proves it is not a FAH issue(which I think we already knew), and it also provides a faster way to check after you make changes(runs in like 10 secs) vs running at minimum 1 FAH frame which for the slower card is around 2+ minutes. There was some other talk in various CUDA forums about heterogeneous cards in the same system but no one had any definate answer.

and from the Cuda Documentation

Page 16, section 3.2 "The use of multiple GPUs as CUDA devices by an application running on a multi-GPU system is only guaranteed to work if theses GPUs are of the same type." The question really is what does this exactly mean for our application: does that mean it may run slower, not at all, and more importantly any solution to get it to work at full speed. It also says "an application" we are actualling using 2 applications" very cloudy indeed

So if anyone is running a mixed (or matched) set, please post up and lets see if we are seeing the same issues. If you need a good reference to what your card should be getting PPD wise here is a direct RS download link to a great little spreadsheet that legoman666 from AT is doing. It gives average speeds for most cards as well as OC settings for anyone unsure of how high your card may be able to clock. So far this link always gets his latest updated version.

Bottom line is if you're running dual cards or want to run dual cards, keep them matched up. They don't have to be running at the same speed OR have the same amount of ram (8800GT 256/8800GT 512 is OK) but they do need to be the same type. If there's free points on the table lets pick um up!

 

[Air Cooled 420]

I have been running a Dual GPU boxen with different GPU, 8800GS and 9600 GSO,
for a bit over week without any of the issues you are having.
Running cards on an older 939 setup with dual 8x PCI-E slots with
X2-3800, WINXP 32 and 1.07 GPU core. 9600 averages 4800ppd, the 8800 GS about
4700ppd, seems pretty rock solid, knock on wood. I haven't bothered moving
to 1.08 after reading your post about the problems with it.

 

[nomad8u]

I have been running a Dual GPU boxen with different GPU, 8800GS and 9600 GSO,
for a bit over week without any of the issues you are having.
Running cards on an older 939 setup with dual 8x PCI-E slots with
X2-3800, WINXP 32 and 1.07 GPU core. 9600 averages 4800ppd, the 8800 GS about
4700ppd, seems pretty rock solid, knock on wood. I haven't bothered moving
to 1.08 after reading your post about the problems with it.

Nice setup and good choice on the cards. You're not having that problem, because they're the same cards. You can compare them here. Same core, same memory, same clocks, same shaders. Different packaging to "extend" the Nvidia line.

The problem is not the 1.08 core. It's a card mismatch. SO just went to bed a short while ago and I *ahem borrowed her 9600GT to pair up with mine. Both are running a solid 4050-4100ppd. That proves it to me for good! I hope she's pleasantly surprised with her new 8800GS in the morning....

 

[Dan-515]

I have this problem as you may have read over at the Folding Forum. I'm pretty pissed off with it to be honest.

Thanks, I've left the 2 GPU clients to run but now I seem to have another problem. It seems only one card is folding to it's full potantial. The PPD are as follows:

GPU 1: 3800
GPU 2: 2200

or

GPU 1: 5300
GPU 2: 2200

So I can see either the GTS (3800 PPD) or the GTX (5300 PPD) is running full speed at one time or another but never at the same time. I was really hoping to let them run and see 3800 and 5300 as simple as that but that's never the case

I don't know why it is picking and choosing which card to go with or why the other is getting these low points. I checked the clocks using GPU-Z and everything is ok. Tried re-installing the drivers but no luck.

Exactly. It was my understanding before I bought my new 9800GTX that you could put several (different) cards in one machine and have them all outputting respective PPD. I thought this was the general consensus.

Now it comes down to it and as people add various cards together they get these odd PPD i.e. I now have a new 9800GTX (G92, 128 SP) and my old 8800GTS 640 (G80, 96 SP), I should be seeing 5000 PPD from the GTX and 3700 PPD from the GTS. I am actually seeing 5000 + 2200 or 3700 + 2200. So depending on which card takes the reigns the other has to limp along on the low PPD.

In the best circumstance I'm getting combined 7200 PPD and at worst 5900. So either a 3500 or 2200 PPD increase when I should have a full 5000 from the newly added GTX.

 

[APOLLO]

Now it comes down to it and as people add various cards together they get these odd PPD i.e. I now have a new 9800GTX (G92, 128 SP) and my old 8800GTS 640 (G80, 96 SP), I should be seeing 5000 PPD from the GTX and 3700 PPD from the GTS. I am actually seeing 5000 + 2200 or 3700 + 2200. So depending on which card takes the reigns the other has to limp along on the low PPD.

In the best circumstance I'm getting combined 7200 PPD and at worst 5900. So either a 3500 or 2200 PPD increase when I should have a full 5000 from the newly added GTX.

I don't mean to disappoint you further but the 9800GTX should be seeing 6000+PPD, from what I recall users reporting with their results. 8800GT cards are producing 5000PPD. So, it's a big drop from the looks of it. Too bad you don't have other motherboards to put the cards in.

 

[nomad8u]

I don't mean to disappoint you further but the 9800GTX should be seeing 6000+PPD, from what I recall users reporting with their results. 8800GT cards are producing 5000PPD. So, it's a big drop from the looks of it. Too bad you don't have other motherboards to put the cards in.

I'm guessing that they might not be OC'd. I've got a GTS (G92 core) that runs a solid 5600 PPD.

 

[nomad8u]

I have this problem as you may have read over at the Folding Forum. I'm pretty pissed off with it to be honest.

The point of this thread wasn't so everyone can get pissed about it. It was to put some data out there for any users that might not be aware of the issue and maybe help guide their purchasing decisions so as to maximize their bang for the buck.

Maybe you can do a swap or trade with someone to get a matched set of cards if you don't have another system to split the cards up in? Just a thought.

 

[Air Cooled 420]

Does your system have a multi-core cpu?
I noticed the same issues when trying
to use 2 GPUs in a single core CPU system. 1 GPU
would fold at top speed the other maybe 2/3rds. My
experience and from what I have read you need a full core
to feed each GPU. When I took out the single core cpu and
put in a dual core both GPUs came back up to top speed.
Good Luck.

 

[Dan-515]

The point of this thread wasn't so everyone can get pissed about it. It was to put some data out there for any users that might not be aware of the issue and maybe help guide their purchasing decisions so as to maximize their bang for the buck.

Maybe you can do a swap or trade with someone to get a matched set of cards if you don't have another system to split the cards up in? Just a thought.

I'm not saying everyone should be pissed I'm saying that I'm pissed as I just bought the 9800GTX before these topics appeared. Of course if it had surfaced a few days earlier I would have thought of some other solution or at least been aware of it. Don't get me wrong I'm glad you have brought the topic here so that all the [H]ard folders don't trip up over something not so obvious.

What Air Cooled 420 said is very interesting, a 9600GSO averaging 4800 PPD. For what I could get for the 9800GTX I could almost have 2x GSOs instead.

 

[Dan-515]

I don't mean to disappoint you further but the 9800GTX should be seeing 6000+PPD, from what I recall users reporting with their results. 8800GT cards are producing 5000PPD. So, it's a big drop from the looks of it. Too bad you don't have other motherboards to put the cards in.

Both my cards are stock (and not factory OC'd), I've been running at around 5000-5300 PPD on the 9800GTX which I'm happy with, I thought a 260 would be around 6000+ or so?

I was and still am tempted to buy a cheap uATX board and a Q6600, as soon as I have a few pennies.

 

[nomad8u]

I'm not saying everyone should be pissed I'm saying that I'm pissed as I just bought the 9800GTX before these topics appeared. Of course if it had surfaced a few days earlier I would have thought of some other solution or at least been aware of it. Don't get me wrong I'm glad you have brought the topic here so that all the [H]ard folders don't trip up over something not so obvious.

I know what you mean. I feel your pain... I really do. I had a 9600GT when the GPU client was released and had a second system running just an SMP client. When the frenzy started, I started looking for deals and picked up an 8800GS and an 8800GTS G92 on some awesome deals. I got lucky though because when I discovered the problem, I swapped my 8800GS for my wife's 9600GT to match a pair up. My second system got the 8800GTS so it all sorta worked out in the end. Hopefully you'll get something figured out in the end.

I'd say your most efficient option for what you have is to put the 9800GTX in Slot 1 and your 8800GTS in Slot 2. If they aren't already, OC the shaders on both cards and and you should be good for 9k+ with that combo.

Good Luck!

 

[jws2346]

Thanks for posting that info nomad8u, I'm about to get my first folding capable GPU (I know, late to the party again ) and I've been checking out the Ebay sales where you get two GPU's at the same time. (not necessarily cheaper that way, but at least you know they're somewhat identical and thanks to you I see where that's very important)

What keeps ringin' in my ears is what Mr relic said "I never pay attention to the stupidity posted on the official folding forum.The team fora are a far better resource for folding info". I agree with him 100%. Whether right or wrong I'm going to believe the post of a well known forum member about folding before I believe anything posted on another folding forum.

I also appreciate some lessons in the English language, such as the term "fora" (I dislike the "grammar police", but I'd never heard or seen the word "fora" before, so I looked it up ) Thanks Mr relic

FOLD ON!

 

[APOLLO]

Both my cards are stock (and not factory OC'd), I've been running at around 5000-5300 PPD on the 9800GTX which I'm happy with, I thought a 260 would be around 6000+ or so?

GTX260 is good for 6500PPD and the GTX280 is capable of over 8000PPD. The GT200 cards are rather expensive only to fold with, while the G92 core cards are still the BBFTB. Waiting for more price cuts and the release of the 'GT200b' cards with 55nm dies.

 

[nomad8u]

Thanks for posting that info nomad8u, I'm about to get my first folding capable GPU (I know, late to the party again ) and I've been checking out the Ebay sales where you get two GPU's at the same time. (not necessarily cheaper that way, but at least you know they're somewhat identical and thanks to you I see where that's very important)

Welcome! Just trying to share the wealth as has been done here by so many of the awesome [H]members. Good luck on the cards. I use this really nifty site to compare cards. The important items are the "GPU/core"and "shader" count. Memory amount doesn't really matter (think 8800GT/512 and 8800GT/256) and I don't think it matters that much if the stock clocks are different.

What keeps ringin' in my ears is what Mr relic said "I never pay attention to the stupidity posted on the official folding forum.The team fora are a far better resource for folding info". I agree with him 100%. Whether right or wrong I'm going to believe the post of a well known forum member about folding before I believe anything posted on another folding forum.

I agree. But idiots aside I do keep a fast eye out there on occasion. But my main resources are indeed the team fora. I always verify anything of importance from "that" forum.

I also appreciate some lessons in the English language, such as the term "fora" (I dislike the "grammar police", but I'd never heard or seen the word "fora" before, so I looked it up ) Thanks Mr relic

Yep.. I had to look that one up too. You can always count on relic to keep ya thinking and learning!

 

[Dan-515]

I use this really nifty site to compare cards.

That's a pretty useful site, I didn't realise for the 9800GTX vs. 8800GTS G80 it's 430 vs. 230 GFLOPS

A 280 is 930! Want one.

EDIT: Just looked at the 4850 and 4870, 1000 and 1200 GFLOPS respectively!
I do prefer nVidia but I have to admit that's impressive.

I'd say your most efficient option for what you have is to put the 9800GTX in Slot 1 and your 8800GTS in Slot 2. If they aren't already, OC the shaders on both cards and and you should be good for 9k+ with that combo.

Good Luck!

Got them in that config at the moment, going to try and OC the shaders a bit, does doing this increase temps as much as straight increasing the core clock?

 

[nomad8u]

Does your system have a multi-core cpu?
I noticed the same issues when trying
to use 2 GPUs in a single core CPU system. 1 GPU
would fold at top speed the other maybe 2/3rds. My
experience and from what I have read you need a full core
to feed each GPU. When I took out the single core cpu and
put in a dual core both GPUs came back up to top speed.
Good Luck.

I'm running a Quad with the multi GPU. In XP it looks like you need a dedicated core for the GPU but Vista seems to be able to handle two GPUs on a single core. I think some of the Roos have had some success with multi GPU in XP on a single core but I'm not 100% sure what the exact configs were.

 

[APOLLO]

Got them in that config at the moment, going to try and OC the shaders a bit, does doing this increase temps as much as straight increasing the core clock?

Hard to say but don't bother with the core or memory clocks for folding. They do next to nothing and will only increase heat.

 

[nomad8u]

That's a pretty useful site, I didn't realise for the 9800GTX vs. 8800GTS G80 it's 430 vs. 230 GFLOPS

That's one of my favorite bookmarks. It's been really handy since starting to use the GPU client..

EDIT: Just looked at the 4850 and 4870, 1000 and 1200 GFLOPS respectively!
I do prefer nVidia but I have to admit that's impressive.

They look to be pretty impressive for gaming. If they ever get the GPU core for them optimized to use all their shaders, they ought to rock!

Got them in that config at the moment, going to try and OC the shaders a bit, does doing this increase temps as much as straight increasing the core clock?

Nope increasing the shaders shouldn't increase the temps much at all. You can look at temps in Rivatuner if you're using that for OC'n but I use GPU-Z on the side to verify the OC and have a quick look at temps.

Luck! Get them shaders whining.

 

[Sunin]

I'm not certain about that 4870 rocking, the shaders although more numerous are at much lower clocks... I'm going to guess that in the end it will be pretty close to Nvidia. Thus the reason I began investing in my farm again.

 

[Dan-515]

Just upped the shaders on the 9800GTX from 1688 to 2001, PPD gone up to 5700-5800, upped the 8800GTS from 1188 to 1352, PPD up to 2500. I guess I'm closing in on my expected PPD of 8700 at 8200-8300. Not too shabby.

Guess that's one type of solution, OC more.

 

[bassman]

I have successfully run CUDA applications on two different GPUs. Both cards were running as fast as they each were capable, but I had to modify the amount work sent to each card so one wasn't waiting for the other. Then again, this was on Linux, so that may make my experience irrelevant.

I went searching through CUDA documentation last night and really didn't come up with anything. Although in the SDK there is the MonteCarlo.exe MonteCarloMultiGPU.exe. Interesting that the multiGPU program has a ratio of 1.58 if I divide the iterations of the faster card by the iterations of the slower card, which is the same ratio I get when dividing the PPD of the faster card with the slower. Don't know what use that info is to you guys. Guess it proves it is not a FAH issue(which I think we already knew), and it also provides a faster way to check after you make changes(runs in like 10 secs) vs running at minimum 1 FAH frame which for the slower card is around 2+ minutes. There was some other talk in various CUDA forums about heterogeneous cards in the same system but no one had any definate answer.

I looked at the MonteCarloMultiGPU code after reading, and there is a reason faster GPUs may appear slower when paired with a slower GPU. To know for sure if it's the same issue FaH is showing, I need to know exactly how the ratio from the quote is computed. It's possible FaH has the same issue, but that depends on how FaH is coded.

Since I can't tell if aicjofs has a multicore system, that's another possibility. NVIDIA has always recommended one core per GPU for multiGPU apps.

EDIT: typo, grammar, clarification

 

[wickedld9]

Hey that's my thread from the official forums! Glad it has helped others.

 

[nomad8u]

Good info there bassman.

Thanks to everyone sharing their experiences. Hopefully it'll help someone who is on the fence about a GPU or two purchase or maybe help some who may be bitten with this issue.

Cheers all an Keep Folding On!