![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Not sure where to put this, but I figured I'd post it here since the topic is related to video card benchmarking...
TL;DR: Look at the pretty pictures. All except one are comparisons of two things running the same test. Which thing is better (and by how much) depends on how you look at the data.
Obligatory intro
Computer hardware manufacturers release new products often. To give some indication of relative performance, the products have numbers in their names. The higher the number, the better the thing is, the average customer might think. That is, however, not the case, and this could lead to much confusion, even among hardware enthusiasts. Thankfully, hundreds of hardware websites publish reviews of new products, in which they compare the performance to older ones. This gives the inquiring customer some information to guide their buying decisions.
Each reviewer tests products differently, causing discrepancies among reviews. Review #1 might say X is faster than Y, but Review #2 could claim the opposite. Many variables affect the results, including system hardware, drivers, operating system, benchmarking application, and application settings. All of these are important considerations, but they will not be discussed. Instead, we will focus on the presentation of data.
Game benchmarking should be different
The way data should be presented depends on the application being measured. For example, if you are comparing how long two CPUs take to encode a video at certain settings, then it would be sufficient to report the time. You are measuring time, so you tell your readers the time. It is as simple as it sounds. 124 seconds is 124 seconds to all who care to measure it.
Things get more complicated when you try to compare how fast two CPUs can run a game. The average reviewer would report the average FPS, but that is insufficient information because games are interactive. Readers want to know whether a game is playable. Although playability is subjective, a game is generally playable if it has very little lag or stutters that hinder gameplay. Each person has a different tolerance to stutters, which occur when the FPS dips below his or her threshold FPS. The average FPS is just the average over the entire benchmark run. It tells you nothing about whether the FPS drops below your threshold.
Enter minimum FPS. As the name suggests, this value tells you the lowest FPS reached during the test. If it is greater than your threshold, then you will likely find the game playable. But what if it is not? Does that mean you need better hardware (or lower settings) for the game to be playable?
Averages and minimums are misleading
Consider the two graphs below:
The graphs compare two things, X and Y. Their identities do not matter. The data was gathered with FRAPS. Y has 12.5% higher minimum FPS and 7.7% higher average FPS. Y leads by 3 in minimum FPS and 3.3 in average FPS. Y is faster than X.
Suppose X and Y are video cards. If Y costs 40% more than X, and if the game was enjoyable using either one, then the reviewer might conclude that X has better value since you would have to pay 40% more for about 10% higher FPS. That sounds like a reasonable interpretation, but it isnt.
Like average FPS, minimum FPS provides insufficient information. It does not tell you how often the FPS drops below your threshold. This is important to know because it correlates with the amount of lag you will experience. You can estimate it visually from the FPS versus time graph, but the overlapping curves make it hard. Luckily, we can make it quantitative by examining the FRAPS data.
FRAPS records the FPS every second, so we get one data point every second. To find how often the FPS drops below a given threshold, we can simply count the number of data points that are less than the threshold. Then we divide by the total number of data points, which is the same as the length of the test run in seconds. Mathematically, the equation is:
Since each person has a different threshold FPS, a range of thresholds between 30 and 40 FPS are shown in the graph below:
Suppose your threshold FPS for this game is 34. Looking at the orange bar, you will see that X drops below the threshold nearly twice as often as Y does. Recall that Y was faster than X by only 10% when average and minimum FPS were used. No matter what threshold FPS you choose, Y drops below the threshold much less often than 10% faster would seem to suggest. Given this information, if Y costs 40% more than X, then the prices appear more reasonable. X no longer provides significantly better value than Y.
Some people may wonder, why not graph the percentage of time above the threshold FPS instead? Surely the cards would be closer to the 10% difference figure:
Suddenly, Y doesnt seem to be leading X by that much. This graph is misleading though. Think about it. When the FPS is above your threshold, the game is playable. You dont notice any stutters. You dont blame lag for missing a headshot or losing your entire army. So why do you care how often the FPS is above your threshold? Instead, it is more useful to know how often it is below because that cripples your gameplay. Pretend that you cant aim fluidly when the FPS drops below 34 or so. Would you rather have the FPS below that threshold 14% of the time or 28% of the time?
Although % time below threshold reveals more than minimum and average FPS, it is still not complete because it ignores how far the FPS drops below the threshold. If the threshold is 30, then 18 and 28 FPS are quite different from a playability standpoint. Both of these are treated the same using this method. Clearly, something better is needed.
One way to resolve this is to measure the area enclosed by a horizontal line representing the threshold and the portions of the FPS versus time curve that are below the line:
Since area is a quantity that involves two dimensions (FPS and time), by measuring the size of the red regions, we get a number that represents how severe the lag is and how long it lasts. The area of irregularly shaped regions like these can be calculated using integrals. Here are the results:
The larger the enclosed area, the more severe and frequent the lag we will experience. Hence, a lower number is better. FPS thresholds depend on the person and the game, so a range is represented in the graph above. For any given threshold, the number for Y is much lower than the one for X. The difference between the two is certainly not around 10%. Minimum and average FPS values are misleading, to say the least.
Of course, this last method is not ideal either. It assumes that a FPS decrease of one and one second of lag affect the game experience equally. For example, if the threshold is 30, ten seconds of 29 FPS and one second of 20 FPS would yield the same number. Most people would likely prefer the former. The method also assumes that a decrease from 10 to 9 FPS is the same as going from 30 to 29 FPS. The ways to account for these flaws would introduce many assumptions into the calculations.
Do we really need a threshold?
Everything discussed above assumes the existence of a FPS threshold. The choice is pretty much arbitrary. Why 30 FPS instead of 29 or 31? Does one FPS really make or break the experience? It is difficult to pinpoint the threshold, but doing so gives us a frame of reference. Suppose a review tells you this video card runs some game at 50 FPS. Is that good or bad? You cant answer that question unless you define good and bad, and thats what the threshold is for. Alternatively, you can play the game and find out yourself, but you want to know before purchasing. That is why reviewers tell you about their experience and give you some numbers. The game was playable at 1080p at highest settings is simple but subjective. The average FPS was 50 is unbiased, but it requires a frame of reference. The threshold is required for understanding FPS data, no matter how arbitrary it appears.
And to conclude
Minimum and average FPS values arent good predictors of your gameplay experience. A decrease of 10% in minimum FPS could translate to a 100% increase of time below the threshold FPS. Why, then, do reviews discuss minimums and averages? Reviewers have to strike a balance between accuracy and accessibility. Calculating definite integrals for the FPS versus time curve is accurate, but it is a potential source of confusion. Readers demand honest, accurate information in a format they can understand. Sometimes, sacrifices must be made for clarity. The end result is a blend of simple quantitative data and qualitative evaluations of the gameplay experience. Whether this is sufficient depends on the reader.
Honestly, I wouldn't mind seeing something other than the traditional min/avg/max FPS as far as numbers go. Maybe that's just me.
TL;DR: Look at the pretty pictures. All except one are comparisons of two things running the same test. Which thing is better (and by how much) depends on how you look at the data.
Obligatory intro
Computer hardware manufacturers release new products often. To give some indication of relative performance, the products have numbers in their names. The higher the number, the better the thing is, the average customer might think. That is, however, not the case, and this could lead to much confusion, even among hardware enthusiasts. Thankfully, hundreds of hardware websites publish reviews of new products, in which they compare the performance to older ones. This gives the inquiring customer some information to guide their buying decisions.
Each reviewer tests products differently, causing discrepancies among reviews. Review #1 might say X is faster than Y, but Review #2 could claim the opposite. Many variables affect the results, including system hardware, drivers, operating system, benchmarking application, and application settings. All of these are important considerations, but they will not be discussed. Instead, we will focus on the presentation of data.
Game benchmarking should be different
The way data should be presented depends on the application being measured. For example, if you are comparing how long two CPUs take to encode a video at certain settings, then it would be sufficient to report the time. You are measuring time, so you tell your readers the time. It is as simple as it sounds. 124 seconds is 124 seconds to all who care to measure it.
Things get more complicated when you try to compare how fast two CPUs can run a game. The average reviewer would report the average FPS, but that is insufficient information because games are interactive. Readers want to know whether a game is playable. Although playability is subjective, a game is generally playable if it has very little lag or stutters that hinder gameplay. Each person has a different tolerance to stutters, which occur when the FPS dips below his or her threshold FPS. The average FPS is just the average over the entire benchmark run. It tells you nothing about whether the FPS drops below your threshold.
Enter minimum FPS. As the name suggests, this value tells you the lowest FPS reached during the test. If it is greater than your threshold, then you will likely find the game playable. But what if it is not? Does that mean you need better hardware (or lower settings) for the game to be playable?
Averages and minimums are misleading
Consider the two graphs below:
The graphs compare two things, X and Y. Their identities do not matter. The data was gathered with FRAPS. Y has 12.5% higher minimum FPS and 7.7% higher average FPS. Y leads by 3 in minimum FPS and 3.3 in average FPS. Y is faster than X.
Suppose X and Y are video cards. If Y costs 40% more than X, and if the game was enjoyable using either one, then the reviewer might conclude that X has better value since you would have to pay 40% more for about 10% higher FPS. That sounds like a reasonable interpretation, but it isnt.
Like average FPS, minimum FPS provides insufficient information. It does not tell you how often the FPS drops below your threshold. This is important to know because it correlates with the amount of lag you will experience. You can estimate it visually from the FPS versus time graph, but the overlapping curves make it hard. Luckily, we can make it quantitative by examining the FRAPS data.
FRAPS records the FPS every second, so we get one data point every second. To find how often the FPS drops below a given threshold, we can simply count the number of data points that are less than the threshold. Then we divide by the total number of data points, which is the same as the length of the test run in seconds. Mathematically, the equation is:
Since each person has a different threshold FPS, a range of thresholds between 30 and 40 FPS are shown in the graph below:
Suppose your threshold FPS for this game is 34. Looking at the orange bar, you will see that X drops below the threshold nearly twice as often as Y does. Recall that Y was faster than X by only 10% when average and minimum FPS were used. No matter what threshold FPS you choose, Y drops below the threshold much less often than 10% faster would seem to suggest. Given this information, if Y costs 40% more than X, then the prices appear more reasonable. X no longer provides significantly better value than Y.
Some people may wonder, why not graph the percentage of time above the threshold FPS instead? Surely the cards would be closer to the 10% difference figure:
Suddenly, Y doesnt seem to be leading X by that much. This graph is misleading though. Think about it. When the FPS is above your threshold, the game is playable. You dont notice any stutters. You dont blame lag for missing a headshot or losing your entire army. So why do you care how often the FPS is above your threshold? Instead, it is more useful to know how often it is below because that cripples your gameplay. Pretend that you cant aim fluidly when the FPS drops below 34 or so. Would you rather have the FPS below that threshold 14% of the time or 28% of the time?
Although % time below threshold reveals more than minimum and average FPS, it is still not complete because it ignores how far the FPS drops below the threshold. If the threshold is 30, then 18 and 28 FPS are quite different from a playability standpoint. Both of these are treated the same using this method. Clearly, something better is needed.
One way to resolve this is to measure the area enclosed by a horizontal line representing the threshold and the portions of the FPS versus time curve that are below the line:
Since area is a quantity that involves two dimensions (FPS and time), by measuring the size of the red regions, we get a number that represents how severe the lag is and how long it lasts. The area of irregularly shaped regions like these can be calculated using integrals. Here are the results:
The larger the enclosed area, the more severe and frequent the lag we will experience. Hence, a lower number is better. FPS thresholds depend on the person and the game, so a range is represented in the graph above. For any given threshold, the number for Y is much lower than the one for X. The difference between the two is certainly not around 10%. Minimum and average FPS values are misleading, to say the least.
Of course, this last method is not ideal either. It assumes that a FPS decrease of one and one second of lag affect the game experience equally. For example, if the threshold is 30, ten seconds of 29 FPS and one second of 20 FPS would yield the same number. Most people would likely prefer the former. The method also assumes that a decrease from 10 to 9 FPS is the same as going from 30 to 29 FPS. The ways to account for these flaws would introduce many assumptions into the calculations.
Do we really need a threshold?
Everything discussed above assumes the existence of a FPS threshold. The choice is pretty much arbitrary. Why 30 FPS instead of 29 or 31? Does one FPS really make or break the experience? It is difficult to pinpoint the threshold, but doing so gives us a frame of reference. Suppose a review tells you this video card runs some game at 50 FPS. Is that good or bad? You cant answer that question unless you define good and bad, and thats what the threshold is for. Alternatively, you can play the game and find out yourself, but you want to know before purchasing. That is why reviewers tell you about their experience and give you some numbers. The game was playable at 1080p at highest settings is simple but subjective. The average FPS was 50 is unbiased, but it requires a frame of reference. The threshold is required for understanding FPS data, no matter how arbitrary it appears.
And to conclude
Minimum and average FPS values arent good predictors of your gameplay experience. A decrease of 10% in minimum FPS could translate to a 100% increase of time below the threshold FPS. Why, then, do reviews discuss minimums and averages? Reviewers have to strike a balance between accuracy and accessibility. Calculating definite integrals for the FPS versus time curve is accurate, but it is a potential source of confusion. Readers demand honest, accurate information in a format they can understand. Sometimes, sacrifices must be made for clarity. The end result is a blend of simple quantitative data and qualitative evaluations of the gameplay experience. Whether this is sufficient depends on the reader.
Honestly, I wouldn't mind seeing something other than the traditional min/avg/max FPS as far as numbers go. Maybe that's just me.
