Graphics Card Benchmark Review Data Request

Status
Not open for further replies.
D

davethejackal

Weaksauce
Joined
Oct 13, 2009
Messages
87
Ok, killing time before a hospital appointment so ... I'm trained as a statistician, so I have a couple of suggestions to make, might not appeal but hey, here they are anyway.

1) Bar charts showing the distribution of FPS over a benchmark run. The height of each bar in the graph would show the number of seconds for which a certain FPS was maintained.

e.g. if in a run a card ran between 100 and 102 FPS constantly, ran at 100fps for 20 of the 30 seconds of the benchmark, 101fps for 7 seconds and 102fps for 3 seconds, you'd show a 3 bar graph with bars of height 20, 7 and 3 respectively.

Where the distribution of FPS was wider, of course, you might have to group things in to less precise categories to get meaningful results, e.g. 0-10fps, 11-20fps, 21-30fps... etc

2) Rather than just showing the maximum, minimum and arithmetic mean (average) FPS can you show the median, upper and lower quartile FPS too. Ok so these words won't mean a lot to people but you could phrase them differently and put something like:

75% of the benchmark, this card ran at 25 FPS or greater.
50% of the benchmark, this card ran at 35 FPS or greater.
25% of the benchmark, this card ran at 75 FPS or greater.

or

% of Benchmark---------FPS was at least
75%---------------------------25 FPS
50%---------------------------35 FPS
25%---------------------------75 FPS

Alternatively you could switch things round, have certain standards, say that 'playable' was 40 fps+, 'ok' was 25fps+, 'unplayable' 20fps-, 'twitch gamer', 60fps+ (I'm just making figures up here) and show what percentage of the time a card achieved these standards during a benchmark.

EDIT: At the danger of confusing things further another method might be to show a cumulative bar chart, as well as showing the distribution FPS over time show the amount of time the card achieved a FPS figure or higher. e.g. in the example I gave for the bar charts, there would be 2 bars for 101 FPS (in different colours). one with a height of 7, the number of seconds at which the card ran at 101 FPS, and one with a height of 10, the number of seconds at which it ran at AT LEAST 101FPS (combining the 101 and 102 FPS figures). It would have the advantage of combining both these suggestions in to 1 graph.

EDIT2: Possibly the best solution, rather than showing the data in bar charts you could use and 'area' graph, which shows the same data just differently. Essentially you have a single line drawn on the graph which passes through the points which would have been at the top of each of the bars in the bar chart, the area under the line is shaded in. Same data, more or less, but it is clearer than a bar chart with a zillion bars on it, and it's easier to see the 'big picture, especially when you have two different data sets superimposed on each other using semi transparent shaded areas and a different colours, as per the previous edit suggestion where cumulative figures and distributions would be shown on the same graph.

EDIT3: an even better solution: seperate graphs could be produced for the cumulative and the raw FPS distributions, with data for each of the graphics cards under review shown on each, making comparisons between cards easier.

Yeah I know, overboard, but hey overboard is what us geeks are all about, right? It would certainly give a lot better picture of how a card performs than just the min/max and mean.
 
Last edited:
EDIT: Ack forget it, I've just realised the top graph is the wrong way round and shows the %time each card spends at a certain FPS or lower, if I get time I will repost with a corrected graph. But you get the point.

EDIT2: revised graphs are here: http://img337.imageshack.us/img337/4858/gcardstats.jpg This time the XYG is trailing in Average FPS but spends more time at higher FPS, the difference isn't as much as I'd like but it makes the point, had I more time I'd have dummied up better data. Read below for original post.

Ok... here's what I mean by my third edit. Consider 2 graphics cards, the XYG 260GT 1Gb and the ABC 3900S 1Gb. No they don't exist, I made them up, as did I the figures which I dummied up on a spreadsheet.

In benchmark tests both cards return identical minimum and maximum FPS of 10 and 82 respectively. The average FPS of the ABC is 43.86 and the XYG is 45.16, so it would seem the XYG is the better card here.

However if you look at the graphs showing FPS distribution that I dummied up they look like this:

gcardstats.jpg


(I don't think I chose the best names for the axes/graphs but that's not important.)

The top graph shows the % time of the benchmark each card achieved a certain FPS or higher, the bottom shows the raw distribution of FPS throughout the benchmark.

Here the bottom graph isn't particularly revealing, but might show an unexpected spike in FPS on one of the cards which might point to a particular effect or game sequence the card coped with poorly. However the top chart clearly shows that, despite having a lower FPS the ABC spends much less time at low FPS than the XYG and more time at playable refresh rates. Which then is the better card? It gives a much clearer picture of performance and how it might operate during games. Ok so it might be a little hard for some people to analyse but the reviewer could always include analysis as part of his/her blurb.

It could be argued that the same information could be gleaned from the raw FPS over time graphs Hard OCP currently shows, but it's much clearer this way.

And don't worry just half an hour to go now before I have to go for my appointment so there won't be much more of this :D

btw excellent site, super reviews, you really seem to know your stuff, this is just a suggestion to make it even better ;)
 
Last edited:
That's a very good idea. Although, I think for most people it's more intuitive to look at the [H] graph and look at the points where the game spiked below their desired FPS.

I think what's more interesting is if such phenomenon as you described existed (it probably does). If it does, how often does it happen, and to what degree (Is it negligible? Is it statistically significant that we should start caring about it?)
 
There are cards where the FPS average is lower but the reviewer says 'I'm not sure why but it just seems to give a better game play experience than the figures show'. Even if such graphs are only occasionally revealing, I'd say it's just a rehash of existing data in a more easy to interpret form ... so why not.
 
Last edited:
@Intel_Hydralisk: I'm not sure what you meant by 'statistically significant' but statistical significance is used to measure whether a sample shows a trend or difference in the population as a whole. E.g. if a reviewer were to review 5 ATI 5770s vs 5 nVidia 250s and try to assert that graphics cards based on one chipset were generally better than the other, statistical significance might play a role. I'm not sure it would be appropriate when considering the benchmarks of one particular card playing on particular game on one particular machine. Even then measures such as statistical significance, variance, etc are often not as informative as seeing the distribution of the data with your own eyes, something repeatedly drummed in to me by my old statistics professor (way back in the 80s).
 
We have looked at these methods in the past but will be sticking with the way we do it now as it seems to be more important to the gamer than a bunch of stats.
 
Meh, what does it actually tell them though? Surely the alternative graphs would provide the same information, just presented in a more meaningful way.
 
Thanks for your thoughts, they are noted.
 
Status
Not open for further replies.
Back
Top