• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

[H]'s Video Card Review Method

K

Kido1986

[H]ard|Gawd
Joined
Mar 10, 2003
Messages
1,180
I want to know what you guys think of it. I, personally, do not like it one bit. When you are reviewing a video card, you should set them all to the same settings. It's okay if you want to add a chart similar to [H]'s entire review type but the main comparison should be 1280x1024 4aa/8af (just an example) and a lower setting such as 1024x768 2aa/2af to show direct comparisons.

I understand what they were trying to do. After reading their multiple articles ove their new method, I can see what they are trying to do but I do not think this is the best way to do this. I complete understand they do not want to use "canned" benchmarks such as 3dmark 01/03 for video cards though.

As I'm writing this I'm reading the new x800Pro review and seeing the apple-to-apples comparison in 1 small section. But to be fair to the video card makers, I believe they should make that the main part of the review and have 1 or 2 pages dedicated to the "playibility" benchmarks.

But this is just my thought on this. Let's here your thoughts guys.

Mods: If you don't find this offensive, could you please temporarily pin this?

Thanks everyone.
 
Heheh. Well If I only wrote 2-3 lines as the topic starter, I would seem kinda uneducated just wanting to bitch. I like at least sound intellegent at times :D
 
theres been about 2384279 threads on this already.

usually, the nvidia fanboys hate the new system, whereas the ati fans like them.

im an ati owner at the moment, i would buy a dual-dvi 6800 ultra if they were anywhere to be seen, and didnt have (insert large argument here) crazy power and heat requirements.
 
I love it and don´t understand why it would be a sticky? It has been discussed before...
 
Garage81 said:
usually, the nvidia fanboys hate the new system, whereas the ati fans like them.
Click to expand...

I am a big ATI fan, but not fanboy. I respect Nvidia a bit too but I just think they should all have a fair and even chance.
 
But I don´t understand? Why don´t nVidia have a chance in this kind of benchmarks??? Or did I totally misunderstood you?

Me I really don´t care about synthetic performance I care about in game performance. And benching like this makes it harder for the video card developers to cheat.
 
Oqvist, they are saying that since they put Nvidia cards at lower settings and then claim thats the highest setting the Nvidia will play properly.

I think if thats the fastest the Nvidia will play, set the ATI to that setting test it there to get accurate tests. It doesn't seem fair that they should show ATI @ 16x12 4aa/8af getting 55 fps and Nvidia @ 12x10 4aa/8af getting 75fps. Kinda off for both ATI and Nvidia.
 
The reviews are accurate. when You buy a card do you by it for it's 3d mark score or because it is fast in the games you play at the resolution you like and is reliable. I can say I have a pretty wide range of experience with video cards in general. each has it selling point. Reviews are to assist you in making a decision about a purchase or to inform. the apples to apples comparisons are not always a good thing as they may persuade or dissuade for the wrong reasons. I can tell you I don't play first person shooters so image quality is what I look for not how fast farcry is. I feel the current model works. It may need some tweaking to make most people happy but I think he is going the right direction.
 
The only benchmark that seemed odd in the recent visiontek x800 pro review was bf vietnam, where it seemed they might have been able to kick the resolution up on the 6800 ultra a bit more and still be equal or better to the x800 pros. Also they should have done those apples to apples benchmarks for all the games they used and it wouldn't hurt to use full screen shots saved in png format (unless there is an issue with space and bandwidth). Other than those issues, I thought the review was great.

I also doubt there is any favoritism going on for ATI too, since it was only a year ago that Kyle and Brent were the enemies of Rage3D (ok maybe mainly Kyle :p).
 
Kido1986 said:
As I'm writing this I'm reading the new x800Pro review and seeing the apple-to-apples comparison in 1 small section. But to be fair to the video card makers, I believe they should make that the main part of the review and have 1 or 2 pages dedicated to the "playibility" benchmarks.
Click to expand...

Actually I am not concerned with "being fair to video card makers," I am focussed on showing the average gamer where he is going to get the best bang for his buck.

Any sure, I will sticky this.
 
I kinda worded that bad.

One thing you need to keep in mind is that some cards run better comparitivly as the settings ramp up. In some benchmarks before, the cards can seem no faster than the older models, lets say at 1024 2aa/2af, but then at 1600 4aa/8af it can take very little hit in the performance as the detail goes up.

While the new method could be useful in some situations, I would like to see more apples-to-apples comparison, similar to the CPU reviews, but I understand there are only so many people working there and there is usually a tight time schedule.

Off Topic:
Although many of us have different opinions and views on this and other subject, I guess the differences in us all are what make this community awesome :)
 
Doesn´t all other websites give those apple to apple comparisons? Why should hardocp do it?
 
I dunno about y'all, but when I buy a game, install it on my computer, and start playing it I adjust my settings so that it is playable, usually I crank up all the in-game settings then manipulate the resolution, AA and AF levels until the game runs smooth, and I go enjoy the game.

Since our focus is on gameplay, which exactly encompasses what I posted above, why not review the cards the same way?

Wouldn't this show the gamer which card is able to play a game "better", or the "same" ?

We've been doing this for 8 months now; it does take some getting use to because everyone, including me, has been stuck in the 3dfx mentality. It is OK, there is always a learning curve. We are going to push on doing it this way because we feel it is the right way, especially moving forward into these incredibly programmable video cards.

We will still keep a small apples-to-apples section with no aa / no af and aa/af numbers for those that find benefit in it. But the real meat of our review will be in the highest playable section where we evaluate performance and image quality.

There are plenty of other sites out there that do nothing but apples-to-apples benchmarks, and we encourage you to read as many as you can.
 
Kido1986 said:
I kinda worded that bad.

One thing you need to keep in mind is that some cards run better comparitivly as the settings ramp up. In some benchmarks before, the cards can seem no faster than the older models, lets say at 1024 2aa/2af, but then at 1600 4aa/8af it can take very little hit in the performance as the detail goes up.

While the new method could be useful in some situations, I would like to see more apples-to-apples comparison, similar to the CPU reviews, but I understand there are only so many people working there and there is usually a tight time schedule.

Off Topic:
Although many of us have different opinions and views on this and other subject, I guess the differences in us all are what make this community awesome :)
Click to expand...

Yes, we do keep that in mind, this is why we play games on the card and decide by the best playable IQ is.

We will be adding NoAF and NoAA A2A data in the next evaluation, but probably only three new games.

There are just so many more folks in our community doing it the other way that I see no use in spending resources on the same thing when it is easily found online. There will ALWAYS be sites using 3DMark and canned benchmarks as they are simple and do not take a lot of time to use. When you start evaluating real gameplay, you are talking about doing something that takes a LOT of time and many webmasters do not have the time.
 
Whatever floats your boat. Apples to apples is what I look for. I am smart enough to tell that if a card can crank out 40 fps than the competitor at 1024X768 that I could probably crank it up another notch and still maintain equal frame rates.
 
While I funny understand the new method, and think it is a good idea, I still find myself skipping right to the apples-to-apples benchmarks.

As long as they include several resolutions/settings/etc, I think it's still fair to everyone. It's just an easier way to see how the card being reviewed does, card-for-card.
 
NOTD665 said:
Alright, now I'm a bit confused. When comparing and contrasting cards against one another, don't you want to see which one performs better while maintaining an accpetable IQ level? Comparing them at the same speeds simply doesn't make sense. I liken it to having a car race, but saying you can't go above 60 MPH.

Two cards, same settings, which one makes the games perform and look better?

That's it.

Whatever the case (and I won't even get into the line graphs), you have got the right idea in some other areas. The canned benchmarks simply have to go the way of the dodo on all review sites. I've lost any shreds of respect for FutureMark since the nVidia debacle last year. And finally, everyone's starting to realize that fps is only important if the quality is there as well. Comparison screenshots, AF mappings, whatever gets the job done.
Click to expand...

with games it is

two cards, which one allows you to obtain a higher level of gaming experience*


*gaming experience = performance + IQ
 
Brent_Justice said:
I dunno about y'all, but when I buy a game, install it on my computer, and start playing it I adjust my settings so that it is playable, usually I crank up all the in-game settings then manipulate the resolution, AA and AF levels until the game runs smooth, and I go enjoy the game.

Since our focus is on gameplay, which exactly encompasses what I posted above, why not review the cards the same way?

Wouldn't this show the gamer which card is able to play a game "better", or the "same" ?

We've been doing this for 8 months now; it does take some getting use to because everyone, including me, has been stuck in the 3dfx mentality. It is OK, there is always a learning curve. We are going to push on doing it this way because we feel it is the right way, especially moving forward into these incredibly programmable video cards.

We will still keep a small apples-to-apples section with no aa / no af and aa/af numbers for those that find benefit in it. But the real meat of our review will be in the highest playable section where we evaluate performance and image quality.

There are plenty of other sites out there that do nothing but apples-to-apples benchmarks, and we encourage you to read as many as you can.
Click to expand...



Most everything you stated here I can agree with on some sort of basis Brent. I like the perspective that you are coming at the review with. However, I believe the new graphs aren't as likeable to the public as the previous. Call it a learning curve or whatever you will I just think that is the main focus(well it as atleast for me). :)

Constructive critism, not meant to bash you or your staff at all.

Once again;kudos for the perspective you are taking at the review!!!
 
scientificTHEgreat said:
Most everything you stated here I can agree with on some sort of basis Kyle. I like the perspective that you are coming at the review with. However, I believe the new graphs aren't as likeable to the public as the previous. Call it a learning curve or whatever you will I just think that is the main focus(well it as atleast for me). :)

Constructive critism, not meant to bash you or your staff at all.

Once again;kudos for the perspective you are taking at the review!!!
Click to expand...

i'm not Kyle last time I checked :D
 
Brent_Justice said:
with games it is

two cards, which one allows you to obtain a higher level of gaming experience*

*gaming experience = performance + IQ
Click to expand...

I agree to a certain extent, but there's not a big enough performance difference to justify that formula. Maybe going from a 9700-level card to an X800 you'll see a performance increase (obviously), but an X800 and 6800 are too close to base on pure "gaming experience".
 
Thank you for that very intellegent and insiteful post. That post shall answer all of our questions.
:rolleyes:
 
NOTD665 said:
Two cards, same settings, which one makes the games perform and look better?

That's it.
Click to expand...

The question would be, at what "same settings"?
Max res and AA+AF? lowest res, no AA+AF?
It does make sense to go for the Max settings with any card (since you want to know which one "look better"). If one maxes out at 1280*1024 2xAA+8xAF and the other at 1600*1200 4xAA+16xAF, doesn't that already speak volumes about the cards perfomances?
 
RS3RS said:
While I funny understand the new method, and think it is a good idea, I still find myself skipping right to the apples-to-apples benchmarks.

As long as they include several resolutions/settings/etc, I think it's still fair to everyone. It's just an easier way to see how the card being reviewed does, card-for-card.
Click to expand...


Maybe because while the same video card will exist in different systems, the components are different? ie: Different cpu, mobo, memory frequencies. So you feel that you will get the best "overall" image by going to apples to apples. A 2.4ghz a64 with a x800 will perform different then a 2.8 p4 with the same card. Because people know this, and they know they may not be able to crank up the resolution or pump up the aa/af the way the would like, they go to the apples to apples comparision. I still need to digest what I just typed, but this is what I understand from your post, RS3RS.
 
I actually like it. It is definitely different and I like it. When I'm in the market for a card i want to see how much i can get out of it. When the reviews show which card pulls better at a higher resolution while the other card just can't hang, that gives me an idea of which card i would rather throw my money down for. The way they kind of show how this card seems to tap out at 1280x1024 and how the other card seems to tap out at 1600X1200, well that in a way is kind of apples to apples, it is like watching a drag race, at the last part of the quarter mile, one just takes off while the other seems like it is has one less gear. might be a bit of a flawed analogy, but i just like the way it is presented. Sure it deviates from the norm and it is different that what you are used to seeing, but it is still good and the data is still there it is just presented differently, and i like it.
 
I strongly dislike the method; it makes no sense to me and is even confusing. I was reading the Xtasy X800 review and was confused for a moment when I I looked at the benchmarks and saw the same frame rates. When im buying a video card, I want to know how fast it is in all resolutions, directly compared to another video card. I do not care what the reviewer thinks is the best playable quality and resolution, that’s for the reader to decide based on direct comparisons. And what is the point of having the graphs if the frame rates are all the same? I, like many others skipped to the disappointing apples to apples section, which was far too small. I suggest you either eliminate the highest playable method and just use apples to apples at a variety of different resolutions and image qualities, or make the highest playable section much smaller, and not the main focus of the article. I think you should set up an apples to apples vs. highest playable poll on the main [H] page, and then use whatever method wins. After all, shouldn't you give the readers what they want?
 
I'm glad HardOCP has decided to do something different to show graphic card performance. I can go almost anywhere else and get a more in-depth "apples-to-apples" review. HardOCP is differentiating itself and offering something I can't get anywhere else, and their method helps me in my decision process.
 
I give it a thumbs up actually, I think it reflects their style of not actually giving a # score and letting the reader decide based on their wording what's best.
 
V0ltage said:
I strongly dislike the method; it makes no sense to me and is even confusing. I was reading the Xtasy X800 review and was confused for a moment when I I looked at the benchmarks and saw the same frame rates. When im buying a video card, I want to know how fast it is in all resolutions, directly compared to another video card. I do not care what the reviewer thinks is the best playable quality and resolution, that’s for the reader to decide based on direct comparisons. And what is the point of having the graphs if the frame rates are all the same? I, like many others skipped to the disappointing apples to apples section, which was far too small. I suggest you either eliminate the highest playable method and just use apples to apples at a variety of different resolutions and image qualities, or make the highest playable section much smaller, and not the main focus of the article. I think you should set up an apples to apples vs. highest playable poll on the main [H] page, and then use whatever method wins. After all, shouldn't you give the readers what they want?
Click to expand...

I kinda agree with this post. I think that same res/settings should be used first off - it gives the most useful and comparable data. I think any reader will know the difference between 2fps and 20fps assuming the graphs are created properly. If the reviewer wants to add subjective data on top of the objective data, then that can also be done, but I think it should be in addition to, not instead of.

I have not seen any review on any site that has done all of the following:
A) Provide benchmarks with settings as close as possible at multiple resolutions and AA/AF levels
B) If not using identical filters, shader precisions, AA methods, AF methods, etc, explain to to the reader what the differences are and how they can affect the performance and image quality, good and bad, both in theory and in execution. Since today's cards are so different this explanation portion is very important.
C) Point out any optimizations used (i.e. angle dependent AF, forced brilinear filtering, forced shader replacement)
D) Give a subjective evaluation of the image quality and impact of optimizations, IQ features, etc along with screenshots or movies to backup your assertions
E) Allow the reader to draw their own conclusions based on your primarily objective presentation of data and somewhat smaller subjective analysis of image quality.

If all these things were done in a review I'd be a happy camper, and I think we'd all gain a lot more insight on the inner workings of the cards being reviewed.
 
I for one really like the new method. I didn't at first, as it took more effort to "read." It's simpler to look a series of numbers and pick out the largest. Everybody and their brother is already doing that, why should [H]?

The FPS charts are nice (they could be more readable though). Average FPS on a level is meaningless if the frame rate drops unacceptably low at a critical juncture. A gamer usually wants the best looking game that's stil pleasantly playable. He'll adjust settings until he gets a good compromise.

Here's a suggestion:

Take a level that's familiar with the reviewer, such as "Fort" in FarCry. Adjust settings (for each reviewed card) until the best compromise of pretty vs. performance is reached. Take a couple screenshots of the settings. This would likely be more involved than the current method. Perhaps one architecture renders shadows better and another renders water better. One site (I forget which) has already pointed out that X800 Pros get much better FarCry frame rates with shadows at medium rather than very high.

The reviewer should not play in god mode, as death to poor frame rates will cause him to adjust settings for better game play.

This would let gamers choose what features are more atractive to them. Already we see rendering differences between ATI and Nvidia, and the screenshots let us see which is more pleasing to us.

Oh, and don't worry about fanboys. They will always be pissed when there chosen product does not do as well.

kneht
 
Not everyone's playable framerate coincides with yours. I dislike the reviews. Their new methodology of comparing different settings offers me nothing I couldn't pull out of a full review.
 
obs said:
Not everyone's playable framerate coincides with yours. I dislike the reviews. Their new methodology of comparing different settings offers me nothing I couldn't pull out of a full review.
Click to expand...

True. but if they also included an FPS chart similar to the ones currently used, one could easily still see what was considered acceptable.

FYI, I hate it when FPS drop below 40, so I could tell that with their settings, my X800 Pro was going to be at a lower res than 12x10 in FarCry.

Indeed, the game was greatly enjoyable at 11x8 and 10x7. It is also enjoyable at 16x12 with slightly lower settings. Not quite sure which way I prefer it the best. The more detail and info a review can give me in a presentable format, the better a decision I can make about upcoming purchases.

kneht
 
Raw framerates don't really matter much anymore. Any decent card can outperform your eyes anymore. Now, it's all about the image quality. You can even garner this shift in focus from the video card makers themselves. Gone are the days of boldly stating that they can get 10FPS more than the other guy. It's all about how "good" thier product makes your games look.

I expect other review sites to follow the [H]'s lead in this.
 
spugnor said:
Raw framerates don't really matter much anymore....Gone are the days of boldly stating that they can get 10FPS more than the other guy.
Click to expand...

Agreed. Who cares that the X800 can manage 450 fps whereas the 6800 is "only" managing 432 fps. It's an outdated way to look at a card's performance. Who even plays Q3 anymore?! (Actually I do about once a week, just to keep myself pumped up for Doom3 :) )

What [H]'s reviews tell you is what kind of playable performance you can expect, which I think most people can agree is to have your game setup with as much eye candy as possible while maintaining a 30-60fps framerate. Can you honestly say this isn't the first thing you do when you install a new game: to tweak all the settings possible to achieve this end?

I don't think an "apples-to-apples" comparison has it's place anymore. It did back in the day when you had a choice of three resolutions to run your game at and that was it. Today's games are way more configurable and as well, the optimizations that each card manufacturer are including in their drivers mean that "apples-to-apples" is dead anyways. [H] skirts the whole issue by simply asking, "How does it play?"

Which in the end, is what I want to know before I plop down my $500. Way to go [H]ard|OCP.

My 2 cents.
 
You must log in or register to reply here.
Back
Top