![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Benchmarking the Benchmarks @ [H]
- Thread starter FrgMstr
- Start date
I see what you're saying, but there are many many measurements for weather conditions (temperature, humidity, wind chill, precipitation). This information is useful to us because most of us, when going to weather.com, can look and see "76 F, Sunny, Winds 5mph," and come to the subjective conclusion of a beautiful day. This quantified information is well defined in the context of real life, and leads to what's going on in reality quite directly.
When looking at video card reviews, saying "This card averaged 89fps in a timedemo" tells you much less about subjective things like the smoothness of the gameplay, the quality of the rendering of different areas, texture blending and anti-aliasing quality, etc. There are no good ways to quantify image quality or playability, in the same way you can quantify weather conditions. Even between the red team and the green team, 4xAA can mean completely different things, because each card/driver pair implements AA in a totally different way.
Weather is described in terms of these metrics and standards, just as video games are described in terms of the quality of the experience.
Some great posts in here. I find myself seeing this from a lot of angles. Personally, I like having Anandtech and every other site put their own twist on reviews. That way I can read 4-5 reviews and get different insight to make up my own mind. I like [H] reviews too, but as shown here they can be dissected as well.
I don't know if I'm keen on calling out other websites. Do what you do and don't worry about the other guy. Obviously you have a large audience so it's worked so far. There's merit on how the reviews are done here, let them speak for themselves.
As far as real world goes, the best part I like is when you say X game is totally playable at 25fps or Y game seems sluggish at 50. FPS means a lot, but your observation of how it feels while playing it mean more.
I don't know if I'm keen on calling out other websites. Do what you do and don't worry about the other guy. Obviously you have a large audience so it's worked so far. There's merit on how the reviews are done here, let them speak for themselves.
As far as real world goes, the best part I like is when you say X game is totally playable at 25fps or Y game seems sluggish at 50. FPS means a lot, but your observation of how it feels while playing it mean more.
I thought the article was well done.
Some people are going to like it, some are going to bitch like little girls.
You cant expect everybody to get along, period.
I believe in real gameplay evaluations. I dont have several cards lying around to test, nor do I have the time or energy to play Crysis over and over again. (tough work if you can find it)
So.....I let these guys do the work. I value their opinion, but still look at sites like PCPerspective(seems like an honest chap) and see what they have to say.......if memory serves me correctly PCPerspective uses real world game play as well.
Sometimes the reviews are right on the nose, other times I have some disagreement with the results as they dont match what I see at home when I play. But.....I value the work it takes to put together a review and not just run some canned benchmark.......
if that makes me a [H] fanboy.......so be it.
Some people are going to like it, some are going to bitch like little girls.
You cant expect everybody to get along, period.
I believe in real gameplay evaluations. I dont have several cards lying around to test, nor do I have the time or energy to play Crysis over and over again. (tough work if you can find it)
So.....I let these guys do the work. I value their opinion, but still look at sites like PCPerspective(seems like an honest chap) and see what they have to say.......if memory serves me correctly PCPerspective uses real world game play as well.
Sometimes the reviews are right on the nose, other times I have some disagreement with the results as they dont match what I see at home when I play. But.....I value the work it takes to put together a review and not just run some canned benchmark.......
if that makes me a [H] fanboy.......so be it.
Zoson
Supreme [H]ardness
- Joined
- Aug 4, 2001
- Messages
- 6,113
You bring meaning to timedemos by running them with every single possible setting.
Card A:
Timedemo 1: 89
Tiemdemo 2: 85
Tiemdemo 3: 35
Tiemdemo 4: 75
...
...
Tiemdemo n: nn
Just because you don't play the timedemos doesn't mean that you cannot test every single possible situation your card would face by using timedemos.
Most sites that benchmark test multiple games, and use in-game timedemos that were recorded by their staff of actual gameplay.
Having reliable information that's quantifiable allows you to compare it to other sources. Then, and ONLY then can you make an informed decision. Otherwise it's just taking someones word for it.
The argument that timedemos don't benchmark everything is flawed in one critical way. Your computer should not be limiting your gameplay. Current games are designed for current hardware and soon-to-be hardware. If it is, clearly your review sites arn't helping you build an effective PC, are they?
I believe you are the one that has it backwards. Other websites following [H]'s criteria, method and examples would show the same findings. There would be some margin for error, but that has to be expected in any type of test. It is reproducible.
The scientific community would laugh other sites review processes out of the building due to using faulty testing tools from the get go. When your "tools" don't match up to the results in the "real world" you seem to have a bit of a problem there.
You want to know why other sites hate this type of "review"? It's labor intensive and costs a lot of money. Plus, it doesn't have as many graphs which makes it "look" like it has less content. Most sites don't want to put the time and effort into something if they can get by with something cheaper even if it means their results don't agree with what really happens. They just want to keep running the same useless timedemos over and over because people have been conditioned to think that is what real gameplay is. We aren't in the infant state of 3D accelerators anymore where the differences between cards was only with the framerate. All the different graphical and IQ settings the different cards have make a huge difference. That is what is being tested here, not framerates. 20 graphs of the average FPS of different resolutions and levels of IQ in a timedemo don't tell me a damn thing.
Get over the fact that timedemos don't follow the scientific method. They are flawed as tools and don't show a damn thing about real world gameplay since they don't even take into account things such as AI or some physics. It's a recorded demo, nothing more.
monkeynetman
Limp Gawd
- Joined
- Dec 19, 2005
- Messages
- 492
Question: Can you guys get more apple to apple comparisons up? Since I don't think everybody will have the exact same test setup as you guys, I think it's still fair enough to see how video cards stack up against each other when all settings are the same - rather than you guys just say oh hey these settings are the best with these cards?
It's great that you guys go at it differently, since you basically are saying that synthetic benchmarking can vary its performance over the real world gaming. However, I still like to compare things settings by settings.
It's great that you guys go at it differently, since you basically are saying that synthetic benchmarking can vary its performance over the real world gaming. However, I still like to compare things settings by settings.
Zoson:
Having built in benchmarks doesn't tell the truth..that's ther problem because it's not just [H] that has founded that both AMD and Nvidia cards run better on time demos, but time demos favors AMD more. It's pretty useless since it doesn't tell about game performance. It tells about theoretical performance, not practical.
Thing with this [H]'s method is that testing one card with one game takes pretty much time because they have to repeat these test so many times that it's practically impossible that scores would have taken serious impact on varying conditions on test. I mean it's like flipping coin. If you throw it 4 times most propably you'll get 2 heads and 2 tails..but there's great chance that you'll get significantly different variations. If you flip that coin 100 times this chance gets small.
"Any real scientist after actual information will fight to the death over this process" Give me a break! It starts to seem that you would really think that there would be some real science behind these tests. If it really was science they would also test does this performance apply on very different system settings and they would find out why there's this kind of difference. AND they would have proofs for their results.
I'd admit that there could be different kind of ways to present these scores. I look mostly at Muropaketti's test and it's basically mixture of [H] and most of other sites: They are looking for best playable resolution for reference card (Card being tested) and then meassure how other cards perform with these settings. They usually have 30-40 fps (minimum) as playable.
-----
Having built in benchmarks doesn't tell the truth..that's ther problem because it's not just [H] that has founded that both AMD and Nvidia cards run better on time demos, but time demos favors AMD more. It's pretty useless since it doesn't tell about game performance. It tells about theoretical performance, not practical.
Thing with this [H]'s method is that testing one card with one game takes pretty much time because they have to repeat these test so many times that it's practically impossible that scores would have taken serious impact on varying conditions on test. I mean it's like flipping coin. If you throw it 4 times most propably you'll get 2 heads and 2 tails..but there's great chance that you'll get significantly different variations. If you flip that coin 100 times this chance gets small.
"Any real scientist after actual information will fight to the death over this process" Give me a break! It starts to seem that you would really think that there would be some real science behind these tests. If it really was science they would also test does this performance apply on very different system settings and they would find out why there's this kind of difference. AND they would have proofs for their results.
I'd admit that there could be different kind of ways to present these scores. I look mostly at Muropaketti's test and it's basically mixture of [H] and most of other sites: They are looking for best playable resolution for reference card (Card being tested) and then meassure how other cards perform with these settings. They usually have 30-40 fps (minimum) as playable.
-----
How is that information reliable when parts of the game such as AI are ripped out when you record a timedemo? That's a part of the game and can affect the quality of the gameplay in regards to performance. AI isn't the only thing removed by recording and rerunning a timedemo.
Also, how do you know how that timedemo is running? I don't know of any sites that show graphs with each run through of the timedemo showing the highs and lows in regards to FPS. You get a number called average FPS. Anyone who plays first person shooters will tell you right off the bat that average FPS means jack shit when your FPS plummets causing you to miss a shot or get shot in the process.
Timedemos and the average FPS numbers you get from sites in regards to them don't tell you a damn thing about the hardware. They don't take into account all aspects of gameplay due to their recorded nature. They are flawed and irrelevant. Logic alone should tell you this.
Zoson
Supreme [H]ardness
- Joined
- Aug 4, 2001
- Messages
- 6,113
@Niceone
Who do you guys think design these cards? Scientists and Engineers.
How do you think they quantify the performance of the cards beforehand for games that have yet to be released? How do you think companies can claim 'our new generation performs xx% better than the last!' before it has even had silicon tapped out?
With numbers and benchmarks.
How do I know? I did several years of VLSI(Very Large Scale Integrated Circuit Design).
I sat there with a simulator that showed me my gated clocks. I crunched, by hand, how fast my devices would be on certain hard paths, and then determined its nominal clockspeed.
@ SmokeRngs: As I said earlier, if your computer is croaking on AI and choking your video card that way, your review sites clearly did not effectively inform you well enough to build a computer that doesn't suck, and failed completely and utterly at their mission statement of helping you buy good parts.
Game over. Every single non scientific argument has been completely and utterly destroyed.
Who do you guys think design these cards? Scientists and Engineers.
How do you think they quantify the performance of the cards beforehand for games that have yet to be released? How do you think companies can claim 'our new generation performs xx% better than the last!' before it has even had silicon tapped out?
With numbers and benchmarks.
How do I know? I did several years of VLSI(Very Large Scale Integrated Circuit Design).
I sat there with a simulator that showed me my gated clocks. I crunched, by hand, how fast my devices would be on certain hard paths, and then determined its nominal clockspeed.
@ SmokeRngs: As I said earlier, if your computer is croaking on AI and choking your video card that way, your review sites clearly did not effectively inform you well enough to build a computer that doesn't suck, and failed completely and utterly at their mission statement of helping you buy good parts.
Game over. Every single non scientific argument has been completely and utterly destroyed.
Ummm no?
I still say (with many others) that a time demo tells me nothing about actual game play. It just give me my e-penis length.
I don't care if I get 60 or 35 FPS, does it play smooth? If a large explosion or other large action event happens does the game hiccup? That is ALL I care about.
It doesn't have to be choking on AI to show the difference between gameplay and benchmarks. That's what this whole article was about: differences between benchmarks and gameplay.
I would think that a stock X6800 and 2 gigs of good ram should not be the bottleneck in a video card review, but maybe I'm wrong. The differences are there though, and you can't deny that. In a game like Crysis, you can set your settings on High @1600x1200, run the benchmark, go get coffee, come back and see 40 FPS average. Great, must be good settings. Try and play through the game at those settings and you're not going to have an enjoyable experience. That's what these evaluations were created for.
Now as a PM'd Zoson about, I half agree with his first post, although I do think Kyle and everyone make great reviews for every aspect of the website, I didn't agree with this article. Anandtech didn't call out HardOCP, it was actually posters here making threads like "Kyle vs. Anandtech", and posters at Anandtech that have always been against Kyle's testing methods. So I don't know why this article, which was supposed to be a subjective, mature article turned into a flaming of Derek Wilson. Yes, they benchmarked cutscenes and ran timedemos, but it doesn't have to be a pissing contest. It almost seems like you were pulling a Jakub from Firingsquad with this article, and I'm pretty sure you didn't appreciate that rant Kyle, or maybe you did, I don't know, I wasn't around for it.
Well that's my 2 cents anyways, it was definitely interesting to see the huge difference between the timedemo and running it ingame, etc. Kudos to HardOCP for showing these differences.
You're not seeming to grasp the concept. It doesn't matter what the fucking companies' engineers claim if the results in REAL TESTING aren't there!
We aren't saying that the cards are not benchmarked or produced without science in mind (on the contrary, we all know they are)! We are saying the average gamer doesn't care how they are benchmarked, no matter if you want to call it 'scientific' or not, if it cannot accurately represent a real gaming experience!!!
And that accurate testing is what I'd like to see focused on, not what some engineer has to say about the % increase!
Zoson
Supreme [H]ardness
- Joined
- Aug 4, 2001
- Messages
- 6,113
"REAL TESTING" what is that? As far as I'm concerned, there's only one scientific method in this world. Every school on the planet teaches it. Want to know why? Because it can be effectively applied to any, and all types of information that require quantification.
The only argument worth repeating that I have heard so far is that you remove load from your computer under a timedemo. But even that is weak and paltry. You can easily reproduce load on a cpu with any given number of tools.
I've already approached the 'lets not hiccup' argument in my first post, and I fall into that one too. However, [H]s benchmarks do NOT represent this category. They allow the framerate to dip below 30fps frequently, as can be seen by the line graphs. So you can't even use the settings posted here on [H] and your argument falls to pieces once again.
There's only one scientific method in this world?
I don't really know how to respond to this statement, it just baffles me.
Instead, I won't even address that, but go back to the point: The timedemos DO NOT accurately represent playing the game. The results are there. You can go look yourself. I've tested much of this myself, and while obviously my numbers don't exactly match up with Kyle's, it is evident the numbers in the timedemos, or claimed benchmarks for many games, DO NOT correlate to what kind of performance I actually see (this is what 'real world' testing means...you can call it anything, but it's merely playing the actual game!).
You say my argument falls to pieces, however, I don't even know what yours is!
I'm sorry but I can't locate the part where you actually answer my question. Could you please try to say wheres the real science behind any review site? You started with asking who design these cards..and how do they get their performance numbers that they are telling. Well those numbers that manufacturer tells us is rounded up by engineers and marketing department. Usually those numbers are purely theoretical and..suprise suprise..doesn't really apply in real life. Ironic isn't it?
I'll give you hint: Even if there would be scientist calculating what kind of performance is expected it's not hard science there. It's simply educated guess because they don't know some important variables right away like for example how effective is their driver department in the end.
Self claiming victory is pretty big indicator that one is either low on valid arguments or just simply arrogant on this one. Either way it's not effective argument. Your style is ironically pretty far from being scientifical.
kirbyrj
Fully [H]
- Joined
- Feb 1, 2005
- Messages
- 30,693
I read the article and thought it was pretty interesting. I guess I always wondered why you wouldn't just record your own demo and then "benchmark" that, so it was informative.
I guess I understand what takes place now, but I still think it's hard to say for certain whether or not you exactly duplicated the run on both cards. I guess you can see a trend in the graphs of the "evaluations," and I don't think anyone is trying to deceive someone into buying certain hardware.
Other than that, I still think apples to apples is important. I understand the real world playability argument, but apples to apples is important to see, especially because what is an acceptable real world playability varies between each individual user, and apples to apples is independent of preference.
I guess I understand what takes place now, but I still think it's hard to say for certain whether or not you exactly duplicated the run on both cards. I guess you can see a trend in the graphs of the "evaluations," and I don't think anyone is trying to deceive someone into buying certain hardware.
Other than that, I still think apples to apples is important. I understand the real world playability argument, but apples to apples is important to see, especially because what is an acceptable real world playability varies between each individual user, and apples to apples is independent of preference.
Thing is that [H] is not only one who gets similar differencies between actual game play and time demos so the question right now is why time demos seem to favor AMD.
It's funny how he seem to think that it's strong argument to self claim victory
Reality
[H]ard|Gawd
- Joined
- Feb 16, 2003
- Messages
- 1,937
beat me to it.
supergrover
Limp Gawd
- Joined
- Oct 6, 2004
- Messages
- 169
[H]ardocp does awesome power supply reviews, although I am not sure if evaluating power supplies based on 'user experience' would be so great...Kyle gave bjorn3d crap about them popping in a power supply, slapping a multimeter on a few leads, and just using it a few days and then giving an award based on their experiences. This is kinda how these Video card reviews are done.
[H] has an excellent forum, and the video card reviews are OK if you take them for what they are.
These video card reviews are just a part of the picture when making an informed decision. Attacking Anandtech may not have been the wisest move, as their methods aim to be scientific (and sometimes they fail) and by definition, Kyle's is a subjective review, and they form an opinion by popping in the card and playing games.
Very hard to compare both methods in a scientific manner, but both sites offer something to someone trying to make an informed decision.
Keep up the video card reviews Kyle as I like them... I pop by to see them (maybe even click a banner ad), and I also like head to Anandtech and other places to make an informed decision. You opinion counts, just as if I asked a buddy if he liked a video card for gaming.
Forgot to mention that it was just my opinionbeat me to it.
. It's just that it seems that he doesn't trust that people will believe his arguments and he has to claim victory to get some comfort.
If it makes him feel better I guess.
I don't live in a test tube. If my FPS drop to 25 for 1/2 a second I'm not going to reach into my system and rip my card out. I just don't want to be always running at that speed.
Kyle keeps stressing "Playable Settings". Otherwise it be called "Mean FSP with min integer >30".
[H] is definitely heading in the right direction. Sure, it's not completely polished yet (in the years to come, we''ll have hopefully discovered more accurate ways to benchmark), but it's definitely advancing. And by advancing, I mean heading towards results that can actually be useful for a gamer.
kirbyrj
Fully [H]
- Joined
- Feb 1, 2005
- Messages
- 30,693
Kyle keeps stressing "Playable Settings". Otherwise it be called "Mean FSP with min integer >30".
That is one of my critiques of [H]...playability varies between users. Apples to apples is better IMHO.
Forgot to mention that it was just my opinion
It's just that it seems that he doesn't trust that people will believe his arguments and he has to claim victory to get some comfort.
Let's not start the personal attacks, I don't want this thread closed yet! It can actually be productive!
And we all know it's your opinion, no need to mention it
mach1.9pants
Weaksauce
- Joined
- Nov 23, 2006
- Messages
- 101
I believe you are the one that has it backwards. Other websites following [H]'s criteria, method and examples would show the same findings. There would be some margin for error, but that has to be expected in any type of test. It is reproducible.
The scientific community would laugh other sites review processes out of the building due to using faulty testing tools from the get go. When your "tools" don't match up to the results in the "real world" you seem to have a bit of a problem there.
You want to know why other sites hate this type of "review"? It's labor intensive and costs a lot of money. Plus, it doesn't have as many graphs which makes it "look" like it has less content. Most sites don't want to put the time and effort into something if they can get by with something cheaper even if it means their results don't agree with what really happens. They just want to keep running the same useless timedemos over and over because people have been conditioned to think that is what real gameplay is. We aren't in the infant state of 3D accelerators anymore where the differences between cards was only with the framerate. All the different graphical and IQ settings the different cards have make a huge difference. That is what is being tested here, not framerates. 20 graphs of the average FPS of different resolutions and levels of IQ in a timedemo don't tell me a damn thing.
Get over the fact that timedemos don't follow the scientific method. They are flawed as tools and don't show a damn thing about real world gameplay since they don't even take into account things such as AI or some physics. It's a recorded demo, nothing more.
THIS!
The article is a wake up call for PC sites out there, just relying on canned demos provided by the game just doesn't cut it. If I went to AT and saw the Crysis results there I would think I could game at 1920 resolution with HIGH settings. The fact is you can't. The review is therefore defunct as a review. It does not help one to buy the correct GPU to fit their monitor.
I hope that sites like AT have the balls to fess up that they are wrong and start putting the hard yards in to get complete reviews, even if they included both timedemos AND real time play then we can make up our own minds!
Yes this is true, however they do let you know what they consider "playable"
@ Zoson
I feel that you misinterpret the intent of the reviews here now. I certainly dont come here looking for scientific fact about which card can push the most pixels.
I came here to find out in a daily driving sort of way, what kind of mileage I can expect from the card. Whats the visual quality like, how do the settings and quality compare to other cards on the market.
Too many rusty gamers are stuck in the number rut. I stopped caring how many max fps I could get years ago. I just want to know how much eye candy I can turn on at high resolutions and how it looks. Top tier cards do very well these days in the HD resolution range. Subjective analysis of this is what I want. I want a guy that can actually get his hands on this Ferrari or lotus and tell me what he thinks, because in all reality im not going to be able to do that. I can read the specs on the card and I can pull down data sheets from the vendor sites. Those tell me how fast the card is. I dont need to run suites of synthetic benchmarks to tell me a top of the line video processor is fast.
I need a review like [H] posts. Tell me how the leather feels under your ass, tell me how the shifter feels in your hand. How does it feel in the turns and on the straights. I know the stats are sexy but is this card a Butterface, or a real sex demon? This is what we are get from [H] today.
Take your scientific method and go try to explain to any connoisseur why they shouldn't like something. You cant. Just like taste, gameplay will always be subjective.
That said you need a little Yin and Yang. I feel the [H] provides both where other sites are all Yin.
Peace
I feel that you misinterpret the intent of the reviews here now. I certainly dont come here looking for scientific fact about which card can push the most pixels.
I came here to find out in a daily driving sort of way, what kind of mileage I can expect from the card. Whats the visual quality like, how do the settings and quality compare to other cards on the market.
Too many rusty gamers are stuck in the number rut. I stopped caring how many max fps I could get years ago. I just want to know how much eye candy I can turn on at high resolutions and how it looks. Top tier cards do very well these days in the HD resolution range. Subjective analysis of this is what I want. I want a guy that can actually get his hands on this Ferrari or lotus and tell me what he thinks, because in all reality im not going to be able to do that. I can read the specs on the card and I can pull down data sheets from the vendor sites. Those tell me how fast the card is. I dont need to run suites of synthetic benchmarks to tell me a top of the line video processor is fast.
I need a review like [H] posts. Tell me how the leather feels under your ass, tell me how the shifter feels in your hand. How does it feel in the turns and on the straights. I know the stats are sexy but is this card a Butterface, or a real sex demon? This is what we are get from [H] today.
Take your scientific method and go try to explain to any connoisseur why they shouldn't like something. You cant. Just like taste, gameplay will always be subjective.
That said you need a little Yin and Yang. I feel the [H] provides both where other sites are all Yin.
Peace
They probably won't fess up, but if they at least change their methodology, I'll be satisfied.
Actually it would be better if there were both included. Problem with AtA is that it does tell difference only in one setting.
You're right.. I mean he was just talking about science and scientific methods and I was pointing out that he doesn't seem to know about them..countering his arguments that was on topic. Perhaps there would be better ways to express this.Let's not start the personal attacks, I don't want this thread closed yet! It can actually be productive!
And we all know it's your opinion, no need to mention it
Great Job [H]. Simply confirms what I already knew.
Also those crapping about "true scientific method" need to wake up. Tell me, what is so scientific about a timedemo when the game box says "plays best on ATI", or "plays best on Nvidia?" Lol, have not the developers already gave consideration, (or perhaps been "assisted") to a certain brand of hardware? Scientific indeed...
Ply
Also those crapping about "true scientific method" need to wake up. Tell me, what is so scientific about a timedemo when the game box says "plays best on ATI", or "plays best on Nvidia?" Lol, have not the developers already gave consideration, (or perhaps been "assisted") to a certain brand of hardware? Scientific indeed...
Ply
I don't see why it must be an either or thing.
Theoretical throughputs are just as important in figuring out a card as the current playability in a handful of games chosen by a reviewer.
The biggest problem with [H]'s test is when they are internally invalid.
When after talking about how one setting isn't the same on two different cards the reviewer then proceeds to test in that questionable setting.
Also who evaluates which of the setting maximized are more important?
Most people I know who play(ed) Oblivion question the choices in most of the reviews, especially things like shadows on grass versus actually more grass.
It's subjective, and just like any canned review is not the only source of good info. The best thing to do is to read MORE reviews and get a global picture, which goes contrary to what this article seems to state which is "[H]'s method is the only way, ignore all the other heathens..."
I'll give [H] editorial staff more credibility when they don't contradict themselves so much, especially when it's obvious those contradictions are only in one direction.
Considering all the brouhaha over the power consumption of the R600 versus it's potential only matching or barely exceeding that of the GTX, I'm left wondering why the change in stance when the HD3870X2 consumes more power and also performs below the GTX? This reaction of "At full load it understandably eats up a large amount of power, but still not tremendously compared to the single 8800 GTX." seems contrary to sentiments less than a year ago.
Why the change of heart, a major change in the price of electricity, availability of PSUs, what? Or is this because the authors finally realized that this is unfortunately the case where for the top of the hill people don't care about a few watts, they'll SLi/Xfire together behemoths if that's what's required to get smooth Crysis gameplay on a 30" LCD?
The idea that this is any more relatable to people who don't have the same rig is a little optimistic to say the least.
To think there is one right way of doing this misses the point, and this article is more about defending one's own choices rather than looking to move the entire methodology/model forward.
However coming from a place that like to comment on the validity of other sites openly, it's not surprising the back/forth on this issue.
Theoretical throughputs are just as important in figuring out a card as the current playability in a handful of games chosen by a reviewer.
The biggest problem with [H]'s test is when they are internally invalid.
When after talking about how one setting isn't the same on two different cards the reviewer then proceeds to test in that questionable setting.
Also who evaluates which of the setting maximized are more important?
Most people I know who play(ed) Oblivion question the choices in most of the reviews, especially things like shadows on grass versus actually more grass.
It's subjective, and just like any canned review is not the only source of good info. The best thing to do is to read MORE reviews and get a global picture, which goes contrary to what this article seems to state which is "[H]'s method is the only way, ignore all the other heathens..."
I'll give [H] editorial staff more credibility when they don't contradict themselves so much, especially when it's obvious those contradictions are only in one direction.
Considering all the brouhaha over the power consumption of the R600 versus it's potential only matching or barely exceeding that of the GTX, I'm left wondering why the change in stance when the HD3870X2 consumes more power and also performs below the GTX? This reaction of "At full load it understandably eats up a large amount of power, but still not tremendously compared to the single 8800 GTX." seems contrary to sentiments less than a year ago.
Why the change of heart, a major change in the price of electricity, availability of PSUs, what? Or is this because the authors finally realized that this is unfortunately the case where for the top of the hill people don't care about a few watts, they'll SLi/Xfire together behemoths if that's what's required to get smooth Crysis gameplay on a 30" LCD?
The idea that this is any more relatable to people who don't have the same rig is a little optimistic to say the least.
To think there is one right way of doing this misses the point, and this article is more about defending one's own choices rather than looking to move the entire methodology/model forward.
However coming from a place that like to comment on the validity of other sites openly, it's not surprising the back/forth on this issue.
To be honest, and I don't want to seem ignorant here:
I still don't even see his argument!
I hope you realize that it's not the specific testing that Kyle did here that is the main point, but the methodology OF his testing.
Sure, we could switch settings around (for all those to discuss which are 'most important'), change configurations, play the game differently. Yes, they could do all those things, but they won't. They can't. It would simply take far too long to try every single test for every single configuration, setting, and playstyle.
With that said, again, the point is what they are doing is on the right track, and yields far more accurate results than running a timedemo for those that wish to see results that they may apply to a real gaming experience. It's much more acceptable than basing everything off of an optimized benchmark.
Do you see?
Yoshiyuki Blade
2[H]4U
- Joined
- Aug 17, 2005
- Messages
- 2,881
I think you're overly exaggerating the "popping in the card" part, as the testing methodology is much more complex than that.
It's also a rather weak analogy to compare video cards to power supplies, as theres a lot of *important* information under the hood that goes undetected without the proper equipment.