I was just re-reading through the whole "Firing Squid wastes their Ink" debacle which I thought was amusing at the time, but would like to comment about it now, and about how [H]ardOCP go about reviewing video card performance.
From what I can see the old "canned benchmarks" idea is still the most favourable when it comes to delivering non-bias factual information about how components perform in different scenarios.
The problem [H] has is that it aims for one set frame rate when gaming, and to achieve that tweaks the settings of the game in certain ways, now this has 2 massive problems.
1) The frame rate you aim for is rather low for certain people, and ultimately you will NEVER find one frame rate that is suitable for all your audience, some people simply don't care and would expect 30fps, some people want competative gaming and expect 60+ FPS average.
2) The settings you tweak are to your own preference, some people prefer more AA and would potentially sacrafice in game effects settings to get it, some people prefer pretty explosions and don't notice Jaggies and so might be the opposite, there's a lot of combinations that need to be covered.
You claim that your review methods are superiour to canned benchmarks but all you're doing is providing a subset of data that you'd typically find in a "canned benchmark" and then stamp a bunch of your own opinions ontop and claim that we should all be running X Y and Z settings with this setup.
The user benefits most from an unbias review of a large range of testing which they can interperate themselves and find the subset of results which is most applicable to them. A small subset of results which might only ever apply to a handful of users is practically worthless to anyone with even a slightly different setup or who is after different frame rates or prefers differenet settings.
I personally think (and no offence meant here) this whole new spin on benchmarking/reviewing is just to pull in more readers because you have a new and novel way of doing things, so you create slogans or memorable terms like "canned benchmark" to make original methods appear inferiour, and claim that you're no longer "benchmarking" but testing real world experience which is an obvious attempt to validate your own methods over others.
Again I say, you're providing a subset of data compared to other "canned benchmarks" i can't see how it's possibly better everything else being equal.
However I'd like to stress that "canned benchmarks" aren't always better, they have their problems, mainly that interperating them has to be done with care and readers don't often have enough experience of knowedge to do so without making mistakes.
This wouldn't normally bother me, I'd just go ahead and ignore your video card reviews being that most of the time they're useless to me, but there seems to be this air of superiority that suggests your review method is so head in the clouds better than everyone elses which just makes me squirm in my seat.
How could you improve your reviews...yes I want to offer some advice and not just bitch/moan.
You need to offer benchmarks for those who want higher frame rates, low FPS simply bothers some people and low FPS is a killer for any kind of online play competative play, most CS players expect frame rates steady at 100FPS, most hardcore UT/Quake players expect frame rates similar, sometimes even well into the 100's
You need to stop assuming that the video settings you've picked are the bonefied "best" combination available, they're probably not for a good portion of the players, some people may have very old low res monitors who have to use low res but maybe able to haver potentially higher effects settings, the opposite might be true for somoene with a high res widescreen with a large native resolution.
I think you'll find if you expand on those things you'll be slowly crawling back to the "canned benchmark" route buy offering up more information than any one specific user may need but covering the needs of a larger amount of users.
From what I can see the old "canned benchmarks" idea is still the most favourable when it comes to delivering non-bias factual information about how components perform in different scenarios.
The problem [H] has is that it aims for one set frame rate when gaming, and to achieve that tweaks the settings of the game in certain ways, now this has 2 massive problems.
1) The frame rate you aim for is rather low for certain people, and ultimately you will NEVER find one frame rate that is suitable for all your audience, some people simply don't care and would expect 30fps, some people want competative gaming and expect 60+ FPS average.
2) The settings you tweak are to your own preference, some people prefer more AA and would potentially sacrafice in game effects settings to get it, some people prefer pretty explosions and don't notice Jaggies and so might be the opposite, there's a lot of combinations that need to be covered.
You claim that your review methods are superiour to canned benchmarks but all you're doing is providing a subset of data that you'd typically find in a "canned benchmark" and then stamp a bunch of your own opinions ontop and claim that we should all be running X Y and Z settings with this setup.
The user benefits most from an unbias review of a large range of testing which they can interperate themselves and find the subset of results which is most applicable to them. A small subset of results which might only ever apply to a handful of users is practically worthless to anyone with even a slightly different setup or who is after different frame rates or prefers differenet settings.
I personally think (and no offence meant here) this whole new spin on benchmarking/reviewing is just to pull in more readers because you have a new and novel way of doing things, so you create slogans or memorable terms like "canned benchmark" to make original methods appear inferiour, and claim that you're no longer "benchmarking" but testing real world experience which is an obvious attempt to validate your own methods over others.
Again I say, you're providing a subset of data compared to other "canned benchmarks" i can't see how it's possibly better everything else being equal.
However I'd like to stress that "canned benchmarks" aren't always better, they have their problems, mainly that interperating them has to be done with care and readers don't often have enough experience of knowedge to do so without making mistakes.
This wouldn't normally bother me, I'd just go ahead and ignore your video card reviews being that most of the time they're useless to me, but there seems to be this air of superiority that suggests your review method is so head in the clouds better than everyone elses which just makes me squirm in my seat.
How could you improve your reviews...yes I want to offer some advice and not just bitch/moan.
You need to offer benchmarks for those who want higher frame rates, low FPS simply bothers some people and low FPS is a killer for any kind of online play competative play, most CS players expect frame rates steady at 100FPS, most hardcore UT/Quake players expect frame rates similar, sometimes even well into the 100's
You need to stop assuming that the video settings you've picked are the bonefied "best" combination available, they're probably not for a good portion of the players, some people may have very old low res monitors who have to use low res but maybe able to haver potentially higher effects settings, the opposite might be true for somoene with a high res widescreen with a large native resolution.
I think you'll find if you expand on those things you'll be slowly crawling back to the "canned benchmark" route buy offering up more information than any one specific user may need but covering the needs of a larger amount of users.