Video card reviews: more focus on minimum frame rates!

T

TechonNapkins

Weaksauce
Joined
Oct 20, 2010
Messages
94
I generally like how [H] reviews videocards: showing gameplay benefits at similar performance levels. The graphs also really help illustrate what is going on.

But thanks to [H]'s graphs, a significant difference between the latest Nvidia cards and the latest AMD cards has been made evident (not that this is new knowledge, I just want to keep it in the limelight) Exhibit A:

http://www.hardocp.com/article/2010/10/21/amd_radeon_hd_6870_6850_video_card_review/6

Now, I have GTX460 1GB (OC'd to 820/4000), and a Dell 2209WA monitor running at 76Hz. With maxed settings, plus some light AA, I sometimes dance above 76fps, but usually stay below that (dips into the 50s). Thus, I don't enable VSync, so I avoid jarring frame rate divider drops, and only occasionally suffer tearing when there's no action on screen. Even in this setup, I can feel the difference between the smoother times above 70fps and the more intense times around 50fps. Looking at the charts in the above review, when the ATI's cards crash in performance, they crash HARD (not [H]ard). Overclocking a 6850 to catch up to my card's average performance, my FPS in more intense moments would likely drop well into the 40s, making the experience much more jarring.

Now, in the conclusion of the article:

"We have shown the AMD Radeon HD 6850 to be the true competition for the 1GB GeForce GTX 460. On a pure performance and gameplay experience comparison the AMD Radeon HD 6850 overall allowed a better gameplay experience. There were a couple of games that resulted in an exception, but even then, the video cards were still close. In Civilization V we did find the GeForce GTX 460 1GB video card to produce higher framerates than the Radeon HD 6850, and in turn allowing a higher gameplay setting. In Battlefield: Bad Company 2 both video cards were playable at the same settings and the performance so close it could not be identified without analysis. In the other games though, the AMD Radeon HD 6850 produced higher framerates and a better gameplay experience. These two video cards are clearly battling it out for performance, and an OC card from either one may tip the balance."

This whole conclusion is clearly based on average framerates, and while thats good for benchmarks and bolstering epeen, the subjective experience, which is what [H] is trying to focus on I believe in its video card reviews, is dependent on minimum frame rates (or for matter, the difference in the high/low performance peaks).

On that note, I did find some silver lining that makes the 6870 very attractive to me (again, thanks to [H]'s graphs):

http://www.hardocp.com/article/2010/11/02/amd_morphological_aa_performance_iq_review/3

Look at what Morphological AA does to the graph. It really flattens it out. If for the consistency of performance alone, I'd probably choose MAA over 8x MSAA.

If you play BF:BC2, the 6870 (or OC'd 6850) may be the card to get, since no other AA method can fully anti-alias that game.

Whatever, getting off on a tangent... minimum frame rates... important... comment.
 
Sometimes mins can be meaningless, like with lots of tests of Civ V, the min goes down to zero for every card tested.

What gives the most information is the actual graph of the fps itself--which HardOCP already provides. That way you can see if the card was hugging the minimum a lot or if it was a random spike or something.
 
Plus it's getting harder to know which GPU's are better for certain things - especially since virtually every game is a console port, with frame limitations, caps and not very good support for PC GPU's.
 
You are missing something very important here, In the review you posted above in Civ 5, F1 2010 , and Mafia the 6850 allowed them to add more graphical sxetting such as AA and other effrects, this changes everything because it's no longer apples to apples and changes the min/avg/max frame rates all together. In other scenarios where they are running at exact settings the cards are neck and neck.

Unfortunately Nvidia clocked the GTX 460 way too low. at 675mhz the card is a totally different card from the one you run at your clocks. Your card performs on par with a 6870 and can push much further beyond it at the 930mhz and beyond zone. Bottom line the reviews are subjective but very informative.
 
For what it's worth, I think it would be good in future reviews if the FPS graphs were annotated a bit. For instance, when there's a framerate spike or a divergence between different plots, add a note about what's going on the the game around that time (i.e. maybe there's a lot of level loading from HDD, or a lot of tesselation appeared, etc).
 
As others have said [H] cant compare minimum framerates between cards because of settings. If one card is running 4xAA and the other is 2xAA that 4xAA has a MUCH larger framebuffer and it will effect things.
 
I agree minimum frame rates are absolutely the most important thing when comparing cards in the same class. While its nice to see what extra iq a card buys you, the H method of adjusting image settings per card negates most of the benefit the min graph provides.
 
I like how HardOCP provides performance plots rather than just simple bar graphs. Minimum FPS, like maximum FPS, can be very misleading.
 
Yeah the plots are nice as it lets you see if the minimum frame rates are common or just a rare occurrence. Minimum frame rates are very important though. The busiest and often most difficult parts of games will have the lowest frame rates so you don't want the game to suddenly become stuttery at the most "important" part.
 
It actually appears that the AMD cards are using SSAA there while the NV cards arent. Is that true? That easily explains the tanking performance from time to time if so.
 
All you need to do is look at the included "apples-to-apples" comparison at the bottom of the page. That shows you min FPS when configured with the same settings. The way [H] tests you can't look at min because they look at the highest playable setting for each card, so min FPS aren't necessarily going to correlate.
 
Graphs are definitely a good way to go.

To often you'll have a situation where the game will have a simple brief hiccup or stutter. Even if that only occurs once that is enough to produce a low minimum FPS value, but would be mostly a non-issue during gameplay. If the game is producing moments of low-FPS consistently throughout the benchmark that is a different story, but wouldn't be represented when "minimum FPS" is shown as a single value.
 
GotNoRice said:
Graphs are definitely a good way to go.

To often you'll have a situation where the game will have a simple brief hiccup or stutter. Even if that only occurs once that is enough to produce a low minimum FPS value, but would be mostly a non-issue during gameplay. If the game is producing moments of low-FPS consistently throughout the benchmark that is a different story, but wouldn't be represented when "minimum FPS" is shown as a single value.
Click to expand...


Good point.

To the poster above, are you saying the [H] already includes apple to apple comparisons graphs?
 
R-Type said:
To the poster above, are you saying the [H] already includes apple to apple comparisons graphs?
Click to expand...

This is the apple to apples-to-apples comparison graph from the link the OP posted:
1287676020VD2xGydP4f_6_5.gif


You can almost always find an apples-to-apples comparison at the bottom of the page. This one shows that though all three cards fall to 14fps at some point during the game, the 6870 and 5850 are consistently higher throughout the whole test with the exception of a couple of drops.

As such, I don't really see the OP's point. As far as I'm concerned, [H] do some great testing where you can at any time see what the fps was like. Not only that, but they tell you what they felt was the highest playable settings with that specific card( and computer, granted).
 
Raap said:
This is the apple to apples-to-apples comparison graph from the link the OP posted:
1287676020VD2xGydP4f_6_5.gif


You can almost always find an apples-to-apples comparison at the bottom of the page. This one shows that though all three cards fall to 14fps at some point during the game, the 6870 and 5850 are consistently higher throughout the whole test with the exception of a couple of drops.

As such, I don't really see the OP's point. As far as I'm concerned, [H] do some great testing where you can at any time see what the fps was like. Not only that, but they tell you what they felt was the highest playable settings with that specific card( and computer, granted).
Click to expand...

Just to clarify, the heading on that chart says 'highest playable settings'. Is this truly running the same settings on each card?
 
R-Type said:
Just to clarify, the heading on that chart says 'highest playable settings'. Is this truly running the same settings on each card?
Click to expand...
Actually, it says Highest In-Game Settings. And yes, those cards are all running highest in-game settings at 2560x1600 with 2xAA and 16xAF.
 
Raap said:
This is the apple to apples-to-apples comparison graph from the link the OP posted:
1287676020VD2xGydP4f_6_5.gif


You can almost always find an apples-to-apples comparison at the bottom of the page. This one shows that though all three cards fall to 14fps at some point during the game, the 6870 and 5850 are consistently higher throughout the whole test with the exception of a couple of drops.

As such, I don't really see the OP's point. As far as I'm concerned, [H] do some great testing where you can at any time see what the fps was like. Not only that, but they tell you what they felt was the highest playable settings with that specific card( and computer, granted).
Click to expand...

Specifically this: "all three cards fall to 14fps at some point during the game, the 6870 and 5850 are consistently higher". THAT is my point. Those cards may have a consistently higher fps, ie, average fps, but their fps crashes relatively harder than the Nvidia card. If I can already feel the fps crash now with my GTX460, replacing it with a 6850 would aggravate the problem even further (although, like I mentioned, I am intrigued by the FPS-stabilizing quality of MAA) . Now, in some games the minimum is an anomaly with no relevance to the GPU chosen, but that is the exception rather than the rule.

I suppose to better word my point would be "further emphasis on the subjective experience as showcased by performance extremes shown in FPS graphs", but that's a mouthful. Since the it's the minimum framerate that's been neglected historically, rather than maximum and average, I'm just saying MORE emphasis needs to be placed on the minimum framerate, yet not neglecting the other two. Again, [H]'s graphs deal with this. It's not such a slam against [H] as it is against every other site that doesn't reveal graphs OR minimum FPS... knowing that there is a significant difference between Nvidia and AMD in this generation makes such reviews worthless to me. So, why did I post this on [H] Forums? 'Cause you guys are awesome! I've seen few forums that actually have constructive input and stay on topic.

Give all yourselves a pat on the back!
 
TechonNapkins said:
Specifically this: "all three cards fall to 14fps at some point during the game, the 6870 and 5850 are consistently higher". THAT is my point. Those cards may have a consistently higher fps, ie, average fps, but their fps crashes relatively harder than the Nvidia card. If I can already feel the fps crash now with my GTX460, replacing it with a 6850 would aggravate the problem even further
Click to expand...

Although, I understand more clearly what you mean, even in the apples to apples examples it doesn't hold true here is an example

1287676020VD2xGydP4f_6_5.gif


In the chart you just linked before^^ the cards do not show erratic behavior. When a card is at it's highest value it doesn't drop all the way down to 14 it'll drop to something like 24 so the drops dont feel more erratic. If it went from the top of the chart down to 14 then yes it'll feel worse than if it went from 40 fps down to 14.

Best example of erratic behavior like this would be in the review from today of 5970 running out of vram and tanking the fps. That is below

1289507336SzLWKc8bvR_2_1.gif

above you can see that even from the maximum and average fps peaks it hits the lowest lows and is a testament to what your speaking about.

I really dont think you would notice a difference if you put 6850 and a GTX 460 on a game at apples to apples settings as they are very near the same performance. The 6850 however, will allow you to add a bit more eye candy then you'd have lower minimums etc.. but not crashing down as you see in the second graph above ^^
 
Civ5 is one of those games where minimum FPS is an anomaly and is worthless as a determining factor between cards. In that BC2 graph, though, what you can see is 4 clear peaks in performance with the ATI cards, whereas the Nvidia cards don't have much of a peak. When the performance drops significantly, the Nvidia cards don't crash as hard. The transition isn't as jarring.

Basically, the smoother the graph, the better. This is why I find MAA is so interesting on the new AMD cards.
 
Would it be possible to get the actual numbers for some of these graphs to try and do some more analysis? Or if someone wants to run a similar benchmark themselves and provide numbers.

Not sure what I'd get from it, but it would be fun to see what other statistics/measure could provide more insight.
 
A textbook example of how it should be done:

http://www.hardocp.com/article/2010/12/02/geforce_gtx_580_vs_460_1gb_sli/

This excerpt really nails it:

In Civ 5 we find that the GeForce GTX 460 1GB SLI is producing higher framerates than the GeForce GTX 580. On average the GTX 460 1GB SLI setup is almost 8% faster than the GeForce GTX 580. At first, this seems impressive, but as you dig deeper into the actual gameplay experience returned you will notice that the GeForce GTX 460 1GB SLI has more variance in the framerates than the GeForce GTX 580 does.

Looking at the red line indicating the GeForce GTX 460 1GB SLI, we notice that the framerates fluctuate wildly. It seems that there is a greater change in framerate compared to the GeForce GTX 580. This change in framerate is felt and noticed in the game. As you pan around the map, or zoom in and out and go to different cities there are more pauses in the gameplay. We also felt the framerate moving from slow performance to higher performance as we played with the GeForce GTX 460 1GB SLI. The GeForce GTX 580 had a more consistent framerate, less change in frames per second. With the GeForce GTX 580, the overall framerates may have been lower, but they did not change as much, and as such we didn’t notice performance fluctuating meaning the experience was smoother. Both settings here are playable, but the better gameplay experience in this game was with the GeForce GTX 580.


I <3 [H]
 
I really like how [H] does reviews as is. However, I think I do see sort of what the OP is talking about.

Maybe if a standard deviation or some other measure were included? I think performance is perceived to be "best" whenever the standard deviation is very low. Unfortunately, I think the vid card companies want that max framerate to be awesome, so they likely sacrifice some perceived smoothness for the almighty benchmark.
 
You must log in or register to reply here.
Back
Top