ATI multi-gpu and new 5xxx workunits

[mfl0p]

Test setup:
Multi AMD GPU using xp, core 1.18, older CAL dlls that come with the gpu client, 3x 4850s, a dedicated q6600 core to each card, 680i mobo

3 gpus running 474x project work units was netting 3000ppd per card

1st gpu running new 5651 project, other 2 gpus running old 474x projects shows a major ppd drop on all 3 gpus.

Trying to figure out just what is going on... looking at cpu usage in XP, none of the fahcores are using all 25% of a cpu core.. so I start looking for the bottleneck.. ram is ddr2 800 4-4-3-10 2T, plenty fast.. next started upping FSB keeping the q6600 running as close to stock 2.4ghz as possible while running the ram at a FSB divider that allows exactly 800mhz ram speed. Results:

FSBmhz / CPUghz / GPU activity via CCC (card 1,2,3)
1066 / 2.40 / 78%,85%,86%
1200 / 2.40 / 83%,92%,92%
1400 / 2.45 / 91%,97%,98%
1600 - epic fail for the nvidia 680i mobo

So to sum this up.. Running more than two 48xx AMD GPUS on one box will saturate the FSB with the new 5xxx WUs... I've seen many people mention the cpu ghz as a limiter on small atom WUs, but never anyone mentioning FSB as a bottleneck. Hopefully future core/CAL updates will lower this overhead.

 

[metallicafan]

I just picked up my first 5651 this evening. With the 1.18 core and the new beta dlls I am getting about 700 PPD lower on my 4850 than the older WUs.

 

[mfl0p]

hmm... after letting the box run overnight, it has picked up 3 of the old 474x units and ppd is only 2400-2600 per card. this doesn't make sense.

So what else did I do when messing with the FSB earlier... ah yes, started by resetting all BIOS settings to default, which turned C1E on. Noticed that the multiplier was jumping from x7 to x6 with the 1400mhz FSB while FAH-GPU was running on all 3 cards. Speedstep was off in BIOS, though. Weird.

Switched back to 1066FSB for now with C1E off.. all 3 cards back at 3000ppd.

Will wait and see what happens again when I get a 5xxx unit.

 

[EvilAlchemist]

Don't freak out ... that is all normal
There are a couple of things going on that is affecting your preformance.

First, on the Multi-GPU setup.
The GPU activity % is a known issue.
I have 2 x 4850's in my main system so i have seen this as well in beta testing.
Mike Houston with ATI is going to setup a dual-gpu boxen to study this problem more.
He should have it setup by the end of the week, so I will try and keep you up to date.
The new CAL dll's will help out a little on the CPU preformance.
The client should have some new dll's soon.. that are tweaked just for F@H and those seem to run great these last two weeks I have had them.


You can download and use those to help a little on XP. Vista sees the biggest difference, with cpu usage topping out around 10% per GPU on a Q6600 @ 2.4Ghz

Second, on the WU & PPD. 5651 is a very small work unit.
So are all the others that were just released.
We were running them in testing and I was getting around 2100-2300 ppd on those.
Once the larger Lambda units come out, you will see the PPD go way up.
Core 1.18 has some improvments that not many are aware of.
When the new units come out (No ETA yet) your PPD should see a large increase from what they are now on the 48xx cards

More improvements are to come, since 1.18 is proving to be a very stable core.
Also, they are working on lowering the traffic between the GPU & CPU and that will help even more, but that is a little later down the road.

 

[metallicafan]

Thanks for the info Evil.

You got PM.

 

[jebo_4jc]

evil - any idea how large the "large" ppd increase is?
Are the big WU's in beta testing yet?

Will we see anything near the 6-7k ppd of the GTX260 etc?

 

[sirmonkey1985]

im getting 6200 PPD avg with 5014 wu's on a 8800GT @ 800/1875/1010.. just watched my ppd go up from 6200 to 6300 with the 5016 wu..


btw up until yesterday.. i was averaging 4700-4900 PPD..

 

[EvilAlchemist]

Thanks for the info Evil.

You are welcome.

evil - any idea how large the "large" ppd increase is?

Well, that is a triple sided question - I will attempt to explain.

The benchmark system has a ATI 3850 and the benchmark is 1500 ppd.
The 3850 has 320 SP's and the 4850 has 800 SP's.
So the 48xx series has an edge over the benchmark system.

But, if the ATI core get better & faster , then the benchmark system will adjust the points of the work units to the new benchmark level.

Example:
Unit A takes 4 hours to complete and worth 200 points / 1500 PDD.
If the core get 10% faster and now takes 3 hours to complete that project,
The WU points would be 150 for that work unit because of the 1500 ppd benchmark.
But remember that your card also got the 10% boost so your PPD should stay near the same level, but could also go down a little
It is a constant battle to stay ahead of the benchmark system.

The trick is for ATI to optimizie the full power of the 4850.

Since the 48xx series has 2.5x the number of SP, you can guess that it should get 2.5x the points of the benchmark system, so a rough number is 3750 PPD.
But, that number does not account for better memory bandwith on the 48xx cards, and other improvments that were put into the 48xx generation of cards.

So, using all these factors, we can put a rough guess at around 4000 PPD.
( I don't know for sure yet becuse no large work units have been released yet for testing)

You also have to remember that some WU might run like crap on a 3850 and the 4850 runs them very fast so that number could go even higher.

Now, on to the 4870, which runs at 750mhz vs the 625mhz of the 4850.
This will raise the PPD even more, but your guess is a good as mine there.

There were reports of 3870 (800mhz) users getting 4000ppd on large lambdas on some older cores, and the new cores are even better, with many more improvments down the road. Take that and add 2.5x the SP and you see how high the PPD can go under the right conditions and with future updates.

At this point, we are 1/2 way there, the new Cal dll's helped to drop the CPU usage down to 10%-12% which is great.

Any improvment on the ATI side, will drop the nVidia points by the same %.
Because the work units are benchmarked on the 3850, the faster the ATI side gets, the lower the nVidia PPD will go.
So , if ATI get 20% faster, you will see a 20% drop in nVidia PPD.
In the next few cores from ATI, you will see a 10%-30% increase in speed, so the nVidia cards will see a corsponding 10%-30% drop in PPD.

 

[Xilikon]

Great explanation, EA (not that I'm ignorant, I already know the answer) and this show precisely why the benchmarking model is flawed. When it get more efficient, the ppd should increase in theory but in reality, since they want to keep it on a set value, the end result is a PPD drop. Very often, the optimisations done to the units and cores will stress the card as it use the resources more efficienly (less idle cycles left to be wasted).

However, there is no better solution to fix the issue beside having a way to compute the amount of flops crunched per WU and to compute the amount of points based on it, which is a good way to reward those cards which worked harder and faster. There is some code in the core to compute the amount of flops but Peter Kasson said it need to be tweaked a bit to get a usable value.

 

[EvilAlchemist]

Xilikon, you are in the know with me, so you see most of what i see ..
I hope that Pande can pull that off, but it will take some time to work that out.

You might have opened a can of worms, because now everyone will want to know what the freak we are talking about .... lol ..

I tried to explaine the upcoming PPD raise for ATI and drop for PPD
(acutally been saying that for a while)

but just wait , when it happens, everyone will say ....

NO ONE TOLD US >> THIS IS SHIT .....

 

[metallicafan]

People will just need to remember that Points are useless. Completing WUs are what its all about.

 

[Kendrak]

People will just need to remember that Points are useless. Completing WUs are what its all about.

Some of us however are points junkies..... (shifts weight nervously) And no warning will prepare us for a 40% percent cut in points.

My most recent 9600GSO purchase will cost me 60 after shipping. 5k points now.... maybe 3k points in the near future. Way I see it..... damn cheap points either way.

 

[pgaster]

I got a 9600GSO too and it runs great, but it has the coil whine problem and I can't take it 24/7 so I am running a 4850 in the meantime. I just wish I knew a way to get rid of that high pitch whine. It's hard to figure out which component is the problem.
I also have an 8800GS that has the whine too. Different board layouts and components, but the same high pitched noise.

 

[jebo_4jc]

So, should I download the new CAL Dlls? It sure sounds like like a huge improvement.

I have two 4870s running on an E6850 that would love to run a CPU client, but the current 100% usage due to dual GPU2s obviously precludes that.