Video card review style - comments.

Frosteh

2[H]4U
Joined
Nov 30, 2004
Messages
3,664
I was just re-reading through the whole "Firing Squid wastes their Ink" debacle which I thought was amusing at the time, but would like to comment about it now, and about how [H]ardOCP go about reviewing video card performance.

From what I can see the old "canned benchmarks" idea is still the most favourable when it comes to delivering non-bias factual information about how components perform in different scenarios.

The problem [H] has is that it aims for one set frame rate when gaming, and to achieve that tweaks the settings of the game in certain ways, now this has 2 massive problems.

1) The frame rate you aim for is rather low for certain people, and ultimately you will NEVER find one frame rate that is suitable for all your audience, some people simply don't care and would expect 30fps, some people want competative gaming and expect 60+ FPS average.

2) The settings you tweak are to your own preference, some people prefer more AA and would potentially sacrafice in game effects settings to get it, some people prefer pretty explosions and don't notice Jaggies and so might be the opposite, there's a lot of combinations that need to be covered.

You claim that your review methods are superiour to canned benchmarks but all you're doing is providing a subset of data that you'd typically find in a "canned benchmark" and then stamp a bunch of your own opinions ontop and claim that we should all be running X Y and Z settings with this setup.

The user benefits most from an unbias review of a large range of testing which they can interperate themselves and find the subset of results which is most applicable to them. A small subset of results which might only ever apply to a handful of users is practically worthless to anyone with even a slightly different setup or who is after different frame rates or prefers differenet settings.

I personally think (and no offence meant here) this whole new spin on benchmarking/reviewing is just to pull in more readers because you have a new and novel way of doing things, so you create slogans or memorable terms like "canned benchmark" to make original methods appear inferiour, and claim that you're no longer "benchmarking" but testing real world experience which is an obvious attempt to validate your own methods over others.

Again I say, you're providing a subset of data compared to other "canned benchmarks" i can't see how it's possibly better everything else being equal.

However I'd like to stress that "canned benchmarks" aren't always better, they have their problems, mainly that interperating them has to be done with care and readers don't often have enough experience of knowedge to do so without making mistakes.

This wouldn't normally bother me, I'd just go ahead and ignore your video card reviews being that most of the time they're useless to me, but there seems to be this air of superiority that suggests your review method is so head in the clouds better than everyone elses which just makes me squirm in my seat.

How could you improve your reviews...yes I want to offer some advice and not just bitch/moan.

You need to offer benchmarks for those who want higher frame rates, low FPS simply bothers some people and low FPS is a killer for any kind of online play competative play, most CS players expect frame rates steady at 100FPS, most hardcore UT/Quake players expect frame rates similar, sometimes even well into the 100's

You need to stop assuming that the video settings you've picked are the bonefied "best" combination available, they're probably not for a good portion of the players, some people may have very old low res monitors who have to use low res but maybe able to haver potentially higher effects settings, the opposite might be true for somoene with a high res widescreen with a large native resolution.

I think you'll find if you expand on those things you'll be slowly crawling back to the "canned benchmark" route buy offering up more information than any one specific user may need but covering the needs of a larger amount of users.
 
Very well put, Frosteh. A lot more calmly put and explained than I could have put it.

I think the problem truly comes from the concept of "canned benchmarks". The problem with canned benchmarks has to do with video card manufacturers tuning their cards to perform the best using these benchmarks. This allows them to make their cards and drivers look really good when in reality they are optimizing their systems (hardware and software) to perform great with these benchmarks.

However, the [H] method of tuning selected benchmarks to produce a range of FPS nullifies their results. I find their reviews using these technique as useless (sorry guys, but this is my opinion). Using this methodology, no two cards can be adequately compared to one another. In fact, depending on tweaked parameters, it is possible to make very good cards look as if they are performing like poor performing cards, and the reverse can be true, too

All this said, by myself and Frosteh (and others, I hope), I will admit right now that I no longer us [H] reviews to help me compare and analyze video card performance (video cards are my primary interest on [H], most of the time). All the slipping and sliding of test parameters make it extremely difficult to make sense out of the results.

Being different can be a positive so readers can get a unique perspective on computer components. But the method you've chosen is extreme and seems to be that way just to be different and not for any real (or sound) hardware review purpose. However, I do wish you all the best and hope that you will "come around" before too long.
 
Thank you for your criticism, I have a few words in response.

1) The framerate we aim for is what we consider to be the bare minimum of playability. Sometimes, we get better results, sometimes we do not. It's all up to the card. We like a high quality image, so we are not likely to give a great score to a video card that can not provide a high quality image.

Framerates are *NOT* always an accurate indicator of performance. Sometimes (as with WoW in the ASUS EN7900 GS TOP evaluation), the framerates are good, but the gameplay still suffers. That ASUS card was totally capable of maintaining above 50fps in WoW at 1600x1200 with 4x TR SSAA and 16x AF. However, the game felt jerky and it wasn't smooth and wasn't a pleasant experience. The framerates were excellent, but the gameplay sucked. How could we recommend a a configuration based solely on high framerates if the gameplay wasn't good?

What we do is to load the video card up as much as possible while maintaining playability. It is not 100% objective, but it is also not 100% subjective. It's a numbers game, basically. If video card X can provide framerates Y in game Z, with settings A, B, and C, and video card W can not get as high, then video card X is better than video card W. It does rely on the editors perception, and experience with gaming in general, to find the rough spots and come up with a good configuration.

Nowhere in our evaluations do we say what configurations people *SHOULD* be running. We simply show what we find to be playable on each video card we look at. Obviously we don't look at every single game out there. That is impossible. We have to strike a balance between timely publishing and thorough gaming. Also, if we take too long to evaluate a video card, a new driver can come out, and we'd have to start over. It happens a lot, as a matter of fact. If a new driver comes out while we're testing, and the driver release notes specifically mention the card or class of cards that we're looking at, we will retest with the new driver. More often than not, it doesn't help.

2) The settings we use are naturally not going to appeal to everyone, and they never will. We do try as broadly as we can to appeal to what people want. Aliasing is a big problem, so we do our best to either enable AA or boost resolution. Texture shimmering is also a big problem, but we've found it is easily reduced, almost eliminated in fact, by using AF options in the display driver control panel.

The more settings there are in a game, the more likely our settings are to not please a broad base of people. Oblivion and FSX are prime examples. Some people might not care for traffic in FSX, and prefer to boost scenery instead. Fine. Some people want the aircraft to be as detailed as possible, at the expense of just about everything else. Fine. We can't predict what every single one of our readers wants, so we try to get a broad, somewhat equalized base. Games like WoW and BF2142 are a lot easier. There are only a few options, and they tend not to have a tremendous effect on performance (other than resolution, AA, and AF), so we can usually leave em turned up as high as possible.

We do not currently have any CounterStrike, Unreal Tournament, or Quake game in our testing suite because those games are old. They run fine on just about any video card above ~$100 USD or so.

Some thought may be required to digest what we show you. Things like: If this video card can run WoW at 1600x1200 with 4x AA and 16x AF, I bet it can run 1280x1024 on my 19" monitor even better! If we are evaluating to a higher standard than what your hardware is capable of, it is generally safe to bet that the hardware we are looking it will run better than what you have. I myself have a pair of 19" LCD monitors, so I am always curious how games will run at 1280x1024 (those games that actually support 1280x1024, that is). But I still want to push the video cards further than that, and see what I can make them do.

Again, we do not just assume that everyone will be and/or should be using the configurations we post. Again, we simply post what we find to be playable.

I'm sorry our evaluations are not to your liking. I would encourage you to contact the contributing editors directly with questions or comments, rather than leave broad-based blanket statements here. Our testing doctrine has not caught on everywhere, obviously, and there will probably always be web sites willing to tell you how many 3dmarks a video card gives them with the specific motherboard, processor, memory, and OS tweaks they are using.

Thanks,
Mark.
 
You look too deeply into what the benchmarks settings are. I know what I prefer to run my games at, so if a card can run 12x10 with 16x HQAF and 6xAA, I know that it will tear through the game at 16x12 at the settings I prefer. They try to find the medium point with the most beauty, since a lot of people want beauty more than 100+ FPS.

And hardcore FPS people already know what to buy, they dont look at review sites to decide what to buy. If you want 100+ FPS in something while it still looks beautiful, its pretty simple - top of the line. :)

[H] is the first place I come for reviews, and I trust what is said in those reviews here more than I do at other places. Not to say I don't look elsewhere as well, but the [H] has yet to make a review that has chased me off.
 
FighterAce124 said:
[H] is the first place I come for reviews, and I trust what is said in those reviews here more than I do at other places. Not to say I don't look elsewhere as well, but the [H] has yet to make a review that has chased me off.

Everybody also looks elsewhere for reviews. It's pretty silly to buy a piece of hardware just because some guy on the internet said it was good. I *always* double and triple check the review sites when i'm looking at a part. I also talk to real people in forums, and in real life as much as possible, before I buy something.
 
Think of it this way: Canned bench marks are still needed, and are still out there, but this provides an additional, interesting new perspective.
 
The only good use of canned benchmarks that I can see are for testing and benchmarking future technologies.
 
A few additional comments.

We like a high quality image, so we are not likely to give a great score to a video card that can not provide a high quality image.

This is a large part of the problem, it's too subjective, some people may prefer a card that can kick out a high frame rate with lower settings because they perfer higher frame rates. If you do one very highly detailed review around only a handful of settings which you consider best you miss out all the other types of users who prefer a different balance.

Framerates are *NOT* always an accurate indicator of performance. Sometimes (as with WoW in the ASUS EN7900 GS TOP evaluation), the framerates are good, but the gameplay still suffers. That ASUS card was totally capable of maintaining above 50fps in WoW at 1600x1200 with 4x TR SSAA and 16x AF. However, the game felt jerky and it wasn't smooth and wasn't a pleasant experience. The framerates were excellent, but the gameplay sucked. How could we recommend a a configuration based solely on high framerates if the gameplay wasn't good?

You shouldn't and I wouldn't expect review to miss something important like this out. I see no reason why "canned benchmarks" can't also note performance anomalys also.

What we do is to load the video card up as much as possible while maintaining playability. It is not 100% objective, but it is also not 100% subjective. It's a numbers game, basically. If video card X can provide framerates Y in game Z, with settings A, B, and C, and video card W can not get as high, then video card X is better than video card W.

Again a too narrow band of testing, suitable for person A but not person B, if the same benchmark was run lots of times over lots of potential settings you could gauge what might be an acceptable frame rate for other people. For example if it's reviewed at 1024x768, 1280x960 and 1600x1200 cards might give the results 100fps, 70fps and 40fps. Now if we have 2 people coming along one who wants a 30fps average and one who wants an 80fps average then this review can cater for both of them, at least closer than current [H] reviews do.

Nowhere in our evaluations do we say what configurations people *SHOULD* be running. We simply show what we find to be playable on each video card we look at.

This is implied through what settings you pick for your reviews, the only way the review is helpful is if you happen use/agree with the settings you've picked to do the review in.

Texture shimmering is also a big problem, but we've found it is easily reduced, almost eliminated in fact, by using AF options in the display driver control panel.

A slightly seperate note about this, I did see complaining about Nvidia texture shimmering in reviews on the 7xxx hardware yet when confronted on the forums about why the AF wasn't set to HQ to help elimiate this (which it did quite well for the most part) I got a bit of a bogus response about thats "how its done" rather than any specific reason why this setting was left untouched.

The more settings there are in a game, the more likely our settings are to not please a broad base of people.

Exactly why a "canned benchmark" array of benchmarks across multiple resolutions and multiple quality settings is better.

We do not currently have any CounterStrike, Unreal Tournament, or Quake game in our testing suite because those games are old. They run fine on just about any video card above ~$100 USD or so.

Yeah this was not my point, rather that such games are good examples of games where people demand very high frame rates and people will gladly sacrafice whatever settings necessary to achieve that. Newer games exist in which competative play is just as important and the same group of gamers but also require decent hardware to run at fast speeds.

Some thought may be required to digest what we show you. Things like: If this video card can run WoW at 1600x1200 with 4x AA and 16x AF, I bet it can run 1280x1024 on my 19" monitor even better! If we are evaluating to a higher standard than what your hardware is capable of, it is generally safe to bet that the hardware we are looking it will run better than what you have.

Not being rude, but i'd rather not assume anything, scaling performance down is not so bad because it's safe, but doing the opposite (scaling up) is a danerous game.

I'm sorry our evaluations are not to your liking.

This isn't a concious choice, it's simply a case of I demand better average frame rates in my games than a lot of your reviews delivery, primarily any game I play competativly online. Like any other person i still want good value for money so I push my graphics as far as I can get them while maintaing a good frame rate, however my "good" frame rate is more like 80-100fps average.

Scaling FPS as a measure of performance is a bad idea, FPS is a non-linear method of measuring the performance hit of features such as AF and AA and what applies at lower frame rates varies at higher frame rates.

I would encourage you to contact the contributing editors directly with questions or comments, rather than leave broad-based blanket statements here.

Tempting, but no. I don't think one person asking for sweeping changes is going to get them, but public awreness and an open discussion is far better for creating awareness about potention problems in anything.

Our testing doctrine has not caught on everywhere, obviously, and there will probably always be web sites willing to tell you how many 3dmarks a video card gives them with the specific motherboard, processor, memory, and OS tweaks they are using.

I'm sorry but this is transparent editorial nonsense, not all "canned benchmark" reviews contain 3dmark scores or necessarily any other of the traits above, I'd prefer if we could steer clear of comments like this and stick to a discussion of how [H] are doing things. I do feel a lot of the "why this review method is better" is editorial spin, but thats just my opinion.
 
Mark_Warner said:
The only good use of canned benchmarks that I can see are for testing and benchmarking future technologies.

Do you honestly believe that's their only use?

You would suggest that your review methed is more appropriate for determing the best video card for gaming with frame rates much higher, at least 80FPS, or for somoene who want's to play games using settings which greatly differ from those which you use.
 
Frosteh said:
Do you honestly believe that's their only use?

You would suggest that your review methed is more appropriate for determing the best video card for gaming with frame rates much higher, at least 80FPS, or for somoene who want's to play games using settings which greatly differ from those which you use.

I will correct myself. We also use some canned benchmarks to make sure the hardware we are looking at is performing where it should. Basically making sure that it's not defective: testing fill rates, testing memory bandwidth, that kind of thing. We also use some for testing overclocks. 3dmark on a 2 or 3 hour loop is a pretty good indicator of stability for overclocking. We don't harp on the results, because they aren't necessarily relevant to what we are trying to do. Sometimes, in fact, I have gotten *lower* 3dmark scores on an overclocked video card, even when games are running much faster.

I'm not saying that our evaluation style is useful for people that only care about framerates. Simply put, framerates are less important to us than the "whole" gameplay experience. A lot of people care only about framerates, and while I do think it is immensly stupid, I understand that there have been years and years of "bigger is better" type of reviews out there, contaminating people's perceptions.

By "whole" experience, I refer to image quality, framerates, and playability. Playability is a hard thing to measure, and it is something that has to be experienced. High framerates can be misleading, and obviously you wouldn't want to play the most beautiful game ever at 2 frames per second. You just kindof "know" if something is playable when you get into it. That is why Brent and Kyle look for people with a good eye for that kind of thing.

Gameplay evaluations will never be as objective as canned benchmarks of a squadron of video cards all running the same settings. That much is true. It is equally true, however, that canned benchmarks from a bunch of video cards running the same settings *WILL*NOT* tell you what you can do; how high you can crank it; in any given game, with any given video card.
 
Frosteh said:
I'm sorry but this is transparent editorial nonsense, not all "canned benchmark" reviews contain 3dmark scores or necessarily any other of the traits above, I'd prefer if we could steer clear of comments like this and stick to a discussion of how [H] are doing things. I do feel a lot of the "why this review method is better" is editorial spin, but thats just my opinion.

If you want to talk about our testing methodologies, then you need to understand that what we/I have to say is obviously going to talk about why we/I think [H]'s way is better. Just because you feel it is editorial spin doesn't mean the argument or statement is invalid.
 
FighterAce124 said:
And hardcore FPS people already know what to buy, they dont look at review sites to decide what to buy. If you want 100+ FPS in something while it still looks beautiful, its pretty simple - top of the line. :)
How do hardcore FPS people already know what to buy? Just because something is touted as high-end doesn't mean it's better than the competition, take the GeForce FX series as an example. Review sites are how people get informed, regardless of whether it's your average Joe who knows nothing about computers or the uber geek who pushes his system to the max. The info has to come from somewhere.

I agree with the rest of your post though, I just think this was a pretty silly statement.
Frosteh said:
I'm sorry but this is transparent editorial nonsense, not all "canned benchmark" reviews contain 3dmark scores or necessarily any other of the traits above
3DMark is the textbook definition of a canned benchmark. I don't see how a canned benchmark could possibly give a better representation of a card's performance than actual in-game testing. The way [H] and a few other sites test using actual gameplay shows how the card will perform in an actual real-world situation, as in this is the kind of performance you get when you pop the card into your computer and fire up a few rounds of the game of your choice. You can have your criticisms of the way they measure a card's performance, but there's no argument you can make to prove that canned benchmarks are in any way better than in-game testing to serve as a representation of an actual gameplay session.
 
I wouldn't use the word "problem" ever when it comes to the way [H] reviews video cards. They do it their way, and they keep doing it their way because the audience they have enjoys it quite a bit and credits them accordingly.
The problem here is that you are telling them to change the way they formulate their testing when you have absolutely no way to insure them success while they are content and quite well renounced for doing exactly what they do. There is a reason they are still around.
You're being too much of an extremist to post in the free portion of their forums in my opinion, and I can tell that Mark takes offense to it.
 
I don't take offense at it. I just think that benchmarks are a poor way to guage performance of a video card. It's about more than raw framerates and pixel/polygon pumping. I think it is ignorant in the extreme to assume that a high benchmark score equals a good video card. That used to be the prevailing view, but I hope that we (the PC gaming community) have grown past that point.
 
I really enjoy how these guys do their reviews. They tell you what the max setting you can run the game at and still enjoy the gameplay. This is what I want to know. I already know that if I turn everything to crap settings I can get 100+ fps, but whats the point in that?

As a gamer, I want to have good fps, and a pretty game. I don't need a game that looks like a rendered movie, but I do want it to look as good as possible before I get slowdowns in intense moments / big firefights / crazy explosions.

If they say they can run the game well @ 1600x1200 w/ everything on, I KNOW that it will run even better on my 1280x1024.

I don't want someone to say, run this game on medium settings and you can get 80 fps, but we won't try it on high so just guess.

If you can't see that they offer a better overall perception than the "canned benchmarks", then you need to really read through the review and not just look at the pictures.
 
Back
Top