Changes that [H] Evaluation should make

[alg7_munif]

I've posted this in the Benchmarking the Benchmarks thread but Kyle has suggested that this should be in another thread.

Now we know that [H] real world method would give a lower fps compared to a timedemo however I don't think that we need an article just to tell us that. What the article shows is that the timedemo is faster than real world on both cards and a card gained more performance in a timedemo than another, so big deal right?

Well personally I think that [H] evaluation is more accurate than other method but overall I also think that their evaluation is not very helpful because the lack of data to show how the gpu would scale with resolution, AA and in games settings. They actually did the right thing in their latest article to show how the 8800GTX would scale with high shader setting in Crysis but I think that without a comparison with numbers from HD3870 X2 at high shader setting, it is useless to include the 8800GTX numbers with high shader setting in the first place.

You could argue that to show how the GPUs would scale with different settings in a real world evaluation, [H] crews need to spend a lot of time doing the test so I've suggested before that [H] should just skip doing this graph here and do a couple more of this table here. With more settings used, I think that it would be clearer to us on how well the GPU would scale with different settings. I also think that [H] testing method shouldn't be dead on consistent just to show the scaling in real world because no one would play games that consistent.

If you want to see a consistency, then there are a lot more reviews out there, what I want to see is an accurate testing with different settings to show the strong and weak points of a GPU. I've posted this before but I think that this deserve more that just a "your view is noted" reply from the [H] crew. Without any change being made, they should just reply with "your view is ignored". So [H] readers, what do you think about this?

 

[rennyf77]

sure they could print more playable settings charts with different resolutions, and they have in the past. you could go as far back as a december and see more than one resolution presented in an eval. why they didn't do so with the x2 eval, i don't know. even if they only posted one chart, any discriminating reader can extrapolate what other settings they can enable with a lower resolution. take their cod4 tests. a gtx is able to push 1920x1200 with everything on and 4x transparency multi sampling aa. dropping down to the next widescreen specification at 1680x1050 you can extrapolate that 8x multi sampled aa is possible. in my own experience, the gtx is capable of 8xq super sampling aa in xp sp2 with playable mins of 20 frames. when it comes to high end cards, i don't think they should bother testing at 1200 resolutions, unless that is the only resolution playable with as many features turned on as possible (i.e. the 2900xt in stalker).

 

[Rebel44]

I agree

 

[Gorankar]

Despite the fact my current monitors are all 4:3 crt's, I think that 4:3 resolutions should be dropped for the most part. 5:4,16:9, and 16:10 is what most people are running these days.

I have mixed feelings as far as apples to apples comparisons goes. While I do like to see them, it is a lot harder if not nearly impossible to do apples to apples with modern cards. Ati and Nv cards handle AA, and filtering differently. Supposedly similar settings can have very different impacts visually and performance wise.

Todays tech is making the reviewers work for a living. Back in the day you could pop in a GeForce run a few custom Q3, and UT99, time demos, restore a ghost image, pop in a V3 or whatever and repeat. Then put together a few graphs and make a few comments regarding the performance. Toss it on the net and you were done. Not hard to conceive that one person could do this in a day or two.
How long from hardware in hand til the actual review is ready these days I wonder? With the pressure to publish the moment NDA is lifted, it has got to be tough sometimes.

 

[alg7_munif]

I'm not asking for an apples to apples benchmark but I want to see more numbers from [H] to know how the cards perform with different AA, resolutions and in game settings, not just a single maximum playable settings which sometimes doesn't reflect the settings that people actually use.

 

[Ehren8879]

highest playable settings at discrete resolutions, concluded with a max playable res/setting. Doesn't seem unreasonable.

 

[APOLLO]

I believe the reviewers at [H] limit themselves to certain settings because they play the entire games from beginning to end. This inescapable fact must limit their testing procedure to a certain extent by needing to meet article deadlines, etc., and I can't blame them for this. I had never known video hardware is assessed with such scrutiny by the reviewers here. The result is a more accurate and representative evaluation of the hardware in realistic scenarios. I'll gladly accept the trade off. If they can incorporate extra settings such as additional resolutions in the future, then even better. As long as it doesn't compromise their methodology in any significant fashion.

 

[bluehaze013]

I think [H] should be using highend CPU/Mobo and Ram Combinations to do these tests in order to eliminate any source of bottlenecking. In the past CPU was not a bottleneck with High Res gaming but currently it has been proven that the 3870x2 actually does increase in performance greatly over an 8800GTX with a Quad Core and over 3ghz processor speed. These two cards do not scale the same.

So essentially the [H] review is misleading in painting a picture where they deem the 8800GTX a better buy, sure for someone with an old dual core processor below 3ghz the 8800GTX may perform up to par with the 3870x2 but even in this case the 3870x2 would be the better buy just for future proofing as when you get around to upgrading the CPU your 3870x2 will now outperform the 8800GTX/Ultra for the same or in the case of the Ultra much less money.

If they did this review on an OC'd Quad this whole fiasco could have been avoided.

 

[alg7_munif]

According to this:

So as alluded to above, we go into the game and perform our real world in-game run throughs that give us hard framerate data to present to our readers. We pick a level or portion of the game that we find will give us a worst case scenario in terms of stressing the graphics card. This is a process that takes quite of bit of skill and practice in order to produce data that is comparable across different video cards. The numbers that you see graphed and charted in our evaluations are pulled from actual gameplay. We use FRAPS to record the framerate as we travel through the game. The trick is being able to do it multiple times in “exactly” the same way. You have to liken this to a race car driver that becomes comfortable with his course. With enough practice you can start pulling very tight lap times that are of course comparable. And as any of our video card editors will tell you, practice makes perfect.


Is every “lap” in our real-world gameplay identical? No it is not, but we put a tremendous amount of time and effort into making sure we have data which we think is thoroughly comparable. We think it would be irresponsible to do otherwise. If that is a deal breaker for you, we understand, but please remember that we draw our conclusions about a video card based on real gameplay, not the data graphed on our evaluation pages.

They spent a lot of time to produce a dead on consistent gameplay but it doesn't matter how hard they tried to be consistent, people would still say that their human factor is inconsistent.

I don't think that they need to be that consistent to produce a comparable result. As long as the result is not all over the place, I think that they could save some time by being less consistent and in return they could spend more time to investigate the performance with different settings.

The dead on consistency is not actually needed for the evaluation because even when they are consistent, people would still doubt the human factor and the most important thing in real world is that no one would play a game consistently.

 

[phide]

The dead on consistency is not actually needed for the evaluation because even when they are consistent, people would still doubt the human factor and the most important thing in real world is that no one would play a game consistently.

Agreed. Additionally, many place far too much emphasis is placed on the graphs. The primary points of interest are the observations during extended gameplay performance and not what min, max and average frame rates were achieved during similar yet slightly different recorded run-throughs. No matter how much this is emphasized, there are still going to be users who place supreme emphasis on the graphs, and they'll label such testing 'unscientific'. Lose the graphs, and you no longer have to deal with such complaints (though obviously the complaints don't really matter in the grand scheme of things).

I personally don't feel as if Brent needs to make any changes. I get pretty much exactly what I need to make a solid buying decision as it is. Resolution scaling is probably better observed in timedemo benchmarks anyway, even if the numbers reflected aren't real-world.

 

[rennyf77]

I think [H] should be using highend CPU/Mobo and Ram Combinations to do these tests in order to eliminate any source of bottlenecking. In the past CPU was not a bottleneck with High Res gaming but currently it has been proven that the 3870x2 actually does increase in performance greatly over an 8800GTX with a Quad Core and over 3ghz processor speed. These two cards do not scale the same.

So essentially the [H] review is misleading in painting a picture where they deem the 8800GTX a better buy, sure for someone with an old dual core processor below 3ghz the 8800GTX may perform up to par with the 3870x2 but even in this case the 3870x2 would be the better buy just for future proofing as when you get around to upgrading the CPU your 3870x2 will now outperform the 8800GTX/Ultra for the same or in the case of the Ultra much less money.

If they did this review on an OC'd Quad this whole fiasco could have been avoided.

really? who proved it. and even if they did switch to an overclocked quad, how much of an overclock would be necessary? if they ran a stock clocked quad someone would complain that users who buy $400 video cards overclock a lot, so what number seems feasible that would quiet detractors?

 

[squares]

I think the hardware community as a whole should have a "meeting of the minds" so to call it, to discuss the best methods and techniques on which to evaluate said hardware with. Although competition does breed innovation/success, I think in this case it would be mutually beneficial if all the major sites got together and established some ground rules / guide lines.

 

[rennyf77]

i really don't see that happening, particularly because [H] is so firm in its belief that its testing method best serves its readers. you also have to consider that the giant dookie storm surrounding the disparate methods has garnered each sit its fair share of revenue generating clicks.

 

[BrinNutz]

Instead of graphs and screenshots and whatnot, there could be video of actual gameplay downloadable with the FPS in the corner, similar to FRAPS, but uncapped framerate and no loss of quality.


Now, if there was only some way to do this at all resolutions with different AA levels and such, without the video losing quality and the FPS taking a hit for working this in the background, would be the only "real" way for people to be able to judge if whatever card they want is suitable to their visual likeness.


At least that works out in my tiny little brain....hrmm..



But, as far as [H]'s eval's go...I digg them...literally and figuratively!! =)

 

[squares]

i really don't see that happening, particularly because [H] is so firm in its belief that its testing method best serves its readers. you also have to consider that the giant dookie storm surrounding the disparate methods has garnered each sit its fair share of revenue generating clicks.

Guess you're correct, at the end of the day money is king.