What so many reviews are missing...

CoW]8(0)

[H]ard|Gawd
Joined
Jul 25, 2005
Messages
1,427
I wasn't sure where I would place this thread as it applies to reviews of all hardware components. But I'm betting the "Video Cards" section would be most dependent on it on this.

The biggest thing I feel so many benchmark results are missing is the proper normalization of results. Posting absolute values of results makes comparing data much more difficult and can often give false conclusions.

I also believe reviewers need to learn a little statistics before making proper conclusions. Would a calculation of uncertainty be so much to ask? They could significantly increase the value of their results if they could justify multiple trials of each test.

Well for now, that's all I have to say. I'll see if I can think up more issues.
 
CoW]8(0);1031190765 said:
I wasn't sure where I would place this thread as it applies to reviews of all hardware components. But I'm betting the "Video Cards" section would be most dependent on it on this.

The biggest thing I feel so many benchmark results are missing is the proper normalization of results. Posting absolute values of results makes comparing data much more difficult and can often give false conclusions.

I also believe reviewers need to learn a little statistics before making proper conclusions. Would a calculation of uncertainty be so much to ask? They could significantly increase the value of their results if they could justify multiple trials of each test.

Well for now, that's all I have to say. I'll see if I can think up more issues.

Well they can only test on the grounds that they are gamers/enthusiasts. Thats why they base their conclusions on performance to price stats.
 
But performance is really a relative quantity in terms of price. You're not paying to reach a specific FPS value. You're paying based on how high this FPS value is relative to what's already on the market.

And aside from that, there's also the data interpretation. Without proper statistical analysis, it's hard to come to the conclusion that one graphics card does better than another.

Also, it wouldn't be difficult at all for the reviewer to fix these issues. All it would take is a little bit of math and for the more apathetic individual I believe excel can do the calculations...
 
CoW]8(0);1031191267 said:
But performance is really a relative quantity in terms of price. You're not paying to reach a specific FPS value. You're paying based on how high this FPS value is relative to what's already on the market.

And aside from that, there's also the data interpretation. Without proper statistical analysis, it's hard to come to the conclusion that one graphics card does better than another.

Also, it wouldn't be difficult at all for the reviewer to fix these issues. All it would take is a little bit of math and for the more apathetic individual I believe excel can do the calculations...

If you don't mind, tell me step by step how you would review a given video card, based on your theory.

Also, if you where to apply your theory, do you think the results would really change so much? I'm not sure, but I would bet that the results would still be somewhat similar.
 
Well normalization is very simple. Some reviewers already do it for small pieces of their data (and also visually in their graphs).

Normalization (to 1):
(sample)/(maximum value of the set of samples)

The nice thing this shows is for data like this:
Absolute:
Video Card 1: 21971
Video Card 2: 22438
Video Card 3: 21641

Normalized:
Video Card 1: .98
Video Card 2: 1
Video Card 3: .96

The differences seem a bit higher for the absolute but the actual differences are quite small.

Uncertainty takes a little more work, but it's well worth it to get meaningful results. Basically given multiple trials of the same test, the uncertainty is how much the results vary due to error. For example, many have noticed that their 3dmark score is never consistently the same value.
Uncertainty tells a reader if the %2 difference between Video Card 1 and Video Card 2 even matters. If the uncertainty is >2%, then Video Card 2 does not necessarily perform better than Video Card 1.

IIRC, I believe this is how uncertainty is calculated:
Average the trials +/- (Largest difference between average and trial data)
 
They already test each video card in each game many times at different combinations of settings. Asking them to repeat each of those runs three or four times to get uncertainty would be asking too much, IMO.
 
Well I'm under the impression they already run the tests multiple times to make sure they have stable resuls. And if that were the case, they would only need to perform the simple calculations. Clicking the benchmark button a few more times doesn't seem excessively difficult for better results.

Even good CPU cooling reviews do multiple mounts to make certain their numbers aren't due to bad mounts. And that's a great deal more effort.
 
Not bad suggestions, but it will never happen. Simply because most people who run these tests are well versed in statistical analysis, not to mention many readers would have difficulty understanding the significance of X% uncertainty.
 
CoW]8(0);1031191964 said:
Well I'm under the impression they already run the tests multiple times to make sure they have stable resuls. And if that were the case, they would only need to perform the simple calculations. Clicking the benchmark button a few more times doesn't seem excessively difficult for better results.

Even good CPU cooling reviews do multiple mounts to make certain their numbers aren't due to bad mounts. And that's a great deal more effort.

It depends on who is doing the review. If you read the fine print, some review will say they did several runs of 3dmark or something to get an average, but most will just be a single pass for time sake UNLESS there is something out of the ordinary (e.g. the 2900XT Company of Heros FPS numbers).
 
by the time you figure in the statistics you can still skew the results
 
I'm not following, a reviewer has the ability to skew the results regardless of what calculations he does.
 
CoW]8(0);1031197378 said:
I'm not following, a reviewer has the ability to skew the results regardless of what calculations he does.

Essentially (if I understand correctly), what you're looking for in normalization is say a percentage of performance. E.g. 8800GTX = 100% performance and 8800GTS = 90% of the performance, 8800GTX SLI = 134% performance, etc.
 
Back
Top