![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
davethejackal
Weaksauce
- Joined
- Oct 13, 2009
- Messages
- 87
Well, this is my second post on the subject of improving the meaningfulness of the benchmark statistics provided reviews (the first post here was somewhat prematurely locked, despite some enthusiastic responses).
Here, having had time to think over the ideas I came up with, I now present 'Sustained Performance Charts', with a section at the end on how to calculate them from FRAPS benchmarking log files using Open Office's sCalc (or another similar spreadsheet).
Ok so here (again) is my suggestion for benchmark results, which I've been whoring round a few sites, designed to give a fuller picture of component performance than mere minimum/maximum and average results that are fairly common now.
What the charts show% of time a certain level of performance was achieved during a single benchmark run (not an aggregate over several different benchmarks, though the stats for various components can be shown on the same chart). It's can be adapted for other types of components but graphics cards seem the most obvious, hence the reason I posted the idea here.
To explain, I've made up 2 graphics cards: the XYG 260GT and the ABC 3900S.
I dummied up some benchmark results on scalc. I don't pretend that they're realistic (I just used random number generators and formulae and crazy math functions) but they do make a point and show how these charts might be useful.
The figures show that both complete a benchmark with the same maximum and minimum FPS of 10/82 FPS, the XYG has an average FPS of 47.38 and the ABC 44.91.... so the XYG is the clear winner, right?
Let's take a look at the sustained performance chart:
another version of the chart with data points plotted is shown here, data values might vary slightly as random numbers were used to generate the data
What the chart shows: If you look along the bottom it shows a certain FPS. The left axis shows the % of time that FPS or higher was achieved. So if the line for a specific card passed through the point that corresponds to 30 FPS at the bottom and 90% on the left, it would show that when the benchmark ran, the card achieved 30FPS for 90% of benchmark. In other words, it only dipped below 30FPS 10% of the time. So if the benchmark ran for a total of 100 seconds, for 10 of those seconds it ran at less than 30FPS and for 90 seconds it was producing greater than 30FPS. Make Sense?
We can see that, despite it's higher average FPS, the XYG barely manages to limp above 30FPS 30% of the time, spending 20% of the time below about 19 FPS. Perhaps the problems is with a certain graphical effect or the XYG can't cope when a scene comes particularly crowded as the manufacturer, for some unknown reason, despite giving it 1Gb of expensive superfast DDR5 memory, decided to lump it with a 128-bit data bus. Of course such a design mistake would never happen in reality :sarcastic:, but you get my point.
Conversely the ABC has a much smoother performance curve, maintaining 35+ FPS 90% of the time.
So ... which would you rather game with? Ok, I can hear cries of 'neither' from the hardcore enthusiast '100FPS or die!' gamers, but let's pretend this is a Crysis 4 (TBA) benchmark, so 35+ FPS isn't that bad.
It's not uncommon to read a review where the reviewer rates a card highly, making comments about the 'overall experience' or some such, despite seemingly unimpressive benchmark scores, or denigrates a product for 'choppy gameplay'. These charts provide the data to back up such claims.
CALCULATING USING A SPREADSHEET FROM FRAPS DATA
It would take some extra work but FRAPS will output the time at which each frame in the benchmark was rendered in to a log file in CSV format, which can easily be imported in to a spreadsheet, giving 2 colums, one containing a frame number (which we ignore) and a second showing the time at which each frame was rendered (in ms).
From the times the intervals between frames can be calculated and from these the number of FPS each frame effectively gave rise to stored in a new column on the spreadsheet, using a formula something like:
Interval between frames = <time at which current frame occurred> - <time at which previous frame occurred>
For each row of the FRAPS data after the first.
Another column is then added to store FPS values: FPS = 1000/(interval between frames in ms) for each of the calculated intervals.
Then another column is added, containing a formula to SUMIF all the interval times that are lower or equal it's the interval time and you get the total time spent at each FPS or higher.
The formulae used in each cell in this column would be something like:
=SUMIF(<entire interval data column range>, CONCATENATE("<="; <interval for cell's row>))
Divide this by the duration of the benchmark and you get the % time the benchmark ran at a specific FPS or faster, stored in a final column.
This is plotted on an 'Line' or 'XY' chart, with X as FPS and the % as Y, sorted by X values, using a single line style interpolated with splines or some such (set in scalc's chart options), which gives the sustained performance chart.
A fancier spread sheet might automate the importing of data and allow more than one set of figures to be considered, of course, but this covers the basic ideas.
All from the same raw FPS data HardOCP publishes already as a time series, but rearranged to make it more meaningful to the reader.
Thoughts/comments appreciated.
Here, having had time to think over the ideas I came up with, I now present 'Sustained Performance Charts', with a section at the end on how to calculate them from FRAPS benchmarking log files using Open Office's sCalc (or another similar spreadsheet).
Ok so here (again) is my suggestion for benchmark results, which I've been whoring round a few sites, designed to give a fuller picture of component performance than mere minimum/maximum and average results that are fairly common now.
What the charts show% of time a certain level of performance was achieved during a single benchmark run (not an aggregate over several different benchmarks, though the stats for various components can be shown on the same chart). It's can be adapted for other types of components but graphics cards seem the most obvious, hence the reason I posted the idea here.
To explain, I've made up 2 graphics cards: the XYG 260GT and the ABC 3900S.
I dummied up some benchmark results on scalc. I don't pretend that they're realistic (I just used random number generators and formulae and crazy math functions) but they do make a point and show how these charts might be useful.
The figures show that both complete a benchmark with the same maximum and minimum FPS of 10/82 FPS, the XYG has an average FPS of 47.38 and the ABC 44.91.... so the XYG is the clear winner, right?
Let's take a look at the sustained performance chart:
another version of the chart with data points plotted is shown here, data values might vary slightly as random numbers were used to generate the data
What the chart shows: If you look along the bottom it shows a certain FPS. The left axis shows the % of time that FPS or higher was achieved. So if the line for a specific card passed through the point that corresponds to 30 FPS at the bottom and 90% on the left, it would show that when the benchmark ran, the card achieved 30FPS for 90% of benchmark. In other words, it only dipped below 30FPS 10% of the time. So if the benchmark ran for a total of 100 seconds, for 10 of those seconds it ran at less than 30FPS and for 90 seconds it was producing greater than 30FPS. Make Sense?
We can see that, despite it's higher average FPS, the XYG barely manages to limp above 30FPS 30% of the time, spending 20% of the time below about 19 FPS. Perhaps the problems is with a certain graphical effect or the XYG can't cope when a scene comes particularly crowded as the manufacturer, for some unknown reason, despite giving it 1Gb of expensive superfast DDR5 memory, decided to lump it with a 128-bit data bus. Of course such a design mistake would never happen in reality :sarcastic:, but you get my point.
Conversely the ABC has a much smoother performance curve, maintaining 35+ FPS 90% of the time.
So ... which would you rather game with? Ok, I can hear cries of 'neither' from the hardcore enthusiast '100FPS or die!' gamers, but let's pretend this is a Crysis 4 (TBA) benchmark, so 35+ FPS isn't that bad.
It's not uncommon to read a review where the reviewer rates a card highly, making comments about the 'overall experience' or some such, despite seemingly unimpressive benchmark scores, or denigrates a product for 'choppy gameplay'. These charts provide the data to back up such claims.
CALCULATING USING A SPREADSHEET FROM FRAPS DATA
It would take some extra work but FRAPS will output the time at which each frame in the benchmark was rendered in to a log file in CSV format, which can easily be imported in to a spreadsheet, giving 2 colums, one containing a frame number (which we ignore) and a second showing the time at which each frame was rendered (in ms).
From the times the intervals between frames can be calculated and from these the number of FPS each frame effectively gave rise to stored in a new column on the spreadsheet, using a formula something like:
Interval between frames = <time at which current frame occurred> - <time at which previous frame occurred>
For each row of the FRAPS data after the first.
Another column is then added to store FPS values: FPS = 1000/(interval between frames in ms) for each of the calculated intervals.
Then another column is added, containing a formula to SUMIF all the interval times that are lower or equal it's the interval time and you get the total time spent at each FPS or higher.
The formulae used in each cell in this column would be something like:
=SUMIF(<entire interval data column range>, CONCATENATE("<="; <interval for cell's row>))
Divide this by the duration of the benchmark and you get the % time the benchmark ran at a specific FPS or faster, stored in a final column.
This is plotted on an 'Line' or 'XY' chart, with X as FPS and the % as Y, sorted by X values, using a single line style interpolated with splines or some such (set in scalc's chart options), which gives the sustained performance chart.
A fancier spread sheet might automate the importing of data and allow more than one set of figures to be considered, of course, but this covers the basic ideas.
All from the same raw FPS data HardOCP publishes already as a time series, but rearranged to make it more meaningful to the reader.
Thoughts/comments appreciated.
Last edited:
