Interesting GPU benchmarking methods article

Replace "Interesting" with "Fairly worthless" in the title and i'd be with ya.
 
pxc said:
Replace "Interesting" with "Fairly worthless" in the title and i'd be with ya.
Click to expand...

Have you done an official review before?

What's your reasoning behind your "fairly worthless" statement?
 
I thought this thread would foster discussion, instead there has been a couple of salvos, and now crickets.

WTH?
 
So they've just found out that time-demo and in game benchmarks don't show the actual performance you see in game. Wonder where I've heard this all before.....
 
Yea, I've read that from Anand and [H], already. It was nice of him to have written the article, but he's really just preaching to the choir. The only people that really needed to read that were lazy reviewers.
 
IRSmurf said:
Seems how you couldn't quantify that,
Click to expand...
Gosh, I almost missed your point from the elephant in the room that you and HC seem to be ignoring. Have you ever read a [H] video card evaluation?

What the article should have been called: "the lazy reviewer's guide to canned benchmarks. it's fast to do and it's good enough!" :p
 
ragejg said:
I thought this thread would foster discussion, instead there has been a couple of salvos, and now crickets.
WTH?
Click to expand...

+1 on that.


Let the discussion begin.


My initial knee-jerk reaction was that HC is so far behind they think they are first. I also figured they were just lazy and trying to justify not always using real world gameplay for their benchmarks.
After reading their article I think they make some compelling points.


I was suprised by their statement that "there are currently a small number of games like DiRT 2 and HawX which incorporate benchmark sequences that accurately recreate in-game scenarios."
I didn't think there were any.


I was even more surprised by their assertion "that games that have the ability to record and play back a timedemo, have for the most part given them accurate results, when timedemos are compared to in-game sequences.
Their comparisons with in-game sequences seem to support this.
http://www.hardwarecanucks.com/foru...g-methods-investigated-fact-vs-fiction-7.html

This is the opposite of what Kyle found in his Benchmarking the benchmarks review of Feb.2008.
http://www.hardocp.com/article/2008/02/11/benchmarking_benchmarks/1
Although Kyle did state in the conclusion of this article that " There is also no doubt that there are some games out there that benchmark perfectly in relation to their real world gameplay. We just don’t know what they are, and quite frankly we don’t care."

I'm not sure if the results would have been different if [H]ard|OCP ran the test today, on a quad core instead of the duo core they used in 2008, or if the Crysis timedemo just doesn't compare well to real world gameplay.

HC's theory of in game timedemos giving accurate results, compared to real world gameplay, is the opposite of what I believed.
I still prefer the way [H]ard|OCP tests with real word gameplay, to eliminate the possibility of a timedemo not being an accurate comparison. I also like the way that they actually give you a recommended setting to max out each cards settings.
Since I play mainly FPS online multiplayer games, I would be interested in knowing if they feel their settings would work well in multiplayer or if they should be dialed back a notch.

I really like the idea of HC giving you a guide to their benchmarking process so you can run them yourself. I'm surprised nobody is trying out the benchmarks and posting their results.
 
DrDoU said:
The [H] is the only review I put any stock in.
Click to expand...

Driverheaven was a good one too. They also used the Real World/Max Playable Settings method but reverted due to the time and financial constraints. That's why I think we won't be see many sites follow [H]'s lead.
 
I don't know about anyone else but I read reviews of video cards to gauge relative performance between different pieces of hardware, I don't care what the actual frame rate is...I have a budget of money I want to spend and I want to find the fastest card for this budget.

How close the frame rate is of a benchmark to actual real game performance is completely and totally irrelevent to me, what matters is the performance delta between cards. It's not as if you're going to see the frame rate, buy the card and then get the same frame rate, the actual frame rate itself is going to be different on your own rig depending on what you have installed and running, what other hardware you're using etc.
 
PrincessFrosty said:
I don't know about anyone else but I read reviews of video cards to gauge relative performance between different pieces of hardware
Click to expand...

But sometimes those canned benchies don't even reflect accurate relative performance (at least for different brands) - Like in that Feb 2008 article also linked above:

"So, had we used even our custom demos to “timedemo benchmark” our two cards in our 3870 X2 evaluation, the 3870 X2 would have enjoyed a “benchmark” advantage over the 8800 GTX when compared to real world gameplay." http://www.hardocp.com/article/2008/02/11/benchmarking_benchmarks/5
 
You must log in or register to reply here.
Back
Top