ATI Radeon HD 3870 X2 @ [H]

Status
Not open for further replies.
I disagree. Hardocp can never reproduce these scores. With canned benchmarks they can.

I agree that using canned bencmarks that come with the game is a bad idea , however using canned benchmarks of your own making where ati / nvidia can't optimise for it then there sholdn't be a problem.

I'll repeat:
If the AI in a game can make different decisions on each run, a timedemo wont cope unless it is scripted.
This defies the whole point as scripted actions wont use the AI engine which does affect performance.

It doesnt matter who canned it.
 
I'll repeat:
If the AI in a game can make different decisions on each run, a timedemo wont cope unless it is scripted.
This defies the whole point as scripted actions wont use the AI engine which does affect performance.

It doesnt matter who canned it.

Sigh ,

Your agreeing with me inside your post but coming to a wrong conclusion.

I'm not talking about the ai itself. I'm talking about the things the ai can choose to do which will change the score compared to a diffrent play through.


For example

Bioshock

Nvidia play throuhg - You walk around find a big daddy and you fight him . You get 50fps

Ati play through - You walk around find a big daddy and fight him , during the fight a second big daddy enters your area and while your fighting the first he becomes enraged also and attacks you. The card gets 40fps

Which card plays better ?
 
I think [H] could do a better job of benchmarking by including the "canned" benchmarks alongside the "real world" benchmarks.

I generally don't like the [H] video card reviews for the simple reason that they don't show apples to apples in their reviews.... The "highest playable settings" thing is not always acurate.

Hmmmm, okay, well let's see why you think so...

I remember quite a few times where certain drivers coupled with certain games would play horrible at certain resolutions but great at higher resolutions. I've also seen quite a few games where turning off or on a certain "feature" would more than double the performance.

Yes, you're right. What you don't seem to understand is that [H]'s method tests and reveals this problem, and the benchmark method does not. [H] tries EVERY resolution and setting before choosing a "best playable," in order to uncover exactly the issues you describe. The other sites don't--they pick "standard" resolutions and settings at the start and use them for everything. Just because [H] only shows the ones that worked the best doesn't mean they didn't do all the rest. That's how they chose which settings were best.

There is a LOT more to doing accurate reviews than "real world" benchmarks.

Like, umm, doing "gonzo world" benchmarks? Just read your statement again and see how hilarious it is. Let me rephrase it for you: "There is a LOT more to finding out the truth than collecting accurate information. You need to gather all the untrue information too."

You also have to take into consideration how long Nvidia has had to get the bugs worked out of their drivers compared to AMD/ATI.

Why? Is having immature drivers somehow a point in ATi's favor?
 
However in canned benchmarks the diffrences are slight.

With hey lets randomly play through this level test things can change drasticly. For instance what if both cards do badly with smoke and in one play through the ai fires off smoke nades or rockets while in the next run through they don't ? Or any number of diffrent combinations .

Have you or have you not done what I describe in the latter part of my post above? Do you or do you not think it represents a valid test of your card's performance?
 
However in canned benchmarks the diffrences are slight.

With hey lets randomly play through this level test things can change drasticly. For instance what if both cards do badly with smoke and in one play through the ai fires off smoke nades or rockets while in the next run through they don't ? Or any number of diffrent combinations .

Are you incapable of noting TRENDS in the graphs? It doesn't matter if card X got slightly better fps than card Y during time N, what matters is the overall trends.Smoke grenades, rockets, etc... everything you have stated only lasts for a couple of seconds, not an entire level. They account for all the peaks and valleys in the graphs. HOWEVER, a slight peak and/or valley doesn't change the overall trend. Playing through an entire level several times, taking roughly the same path and making the same decisions will give an average that is repeatable - try it out yourself. Play the same level in more or less the same way twice and compare your average FPS.

For example, take the Crysis apples-to-apples ( http://enthusiast.hardocp.com/image.html?image=MTIwMTIwNDUyODZqWk1RbU45c1hfM180X2wuZ2lm ). Is the 8800GTX always on top? No, of course not, the card in the lead varies at different points, but overall which card is on top a majority of the time? The 8800GTX - therefore the 8800GTX is better for Crysis. Also, note the peaks and valleys are generally mirrored between the two cards. When one card spikes, the other spikes too, which means that the two run throughs are much closer in what is being rendered than you are willing to admit.

But still, this is a 10 MINUTE run through of a level - various events that you mention will quickly average out and become insignificant over the time period. 20 seconds of marginally lowered FPS due to a smoke grenade over a 10 minute period will have a minimal effect, and it is highly probable that the other card will have experienced the same (if at a different point in time - it doesn't matter).
 
Sigh ,

Your agreeing with me inside your post but coming to a wrong conclusion.

I'm not talking about the ai itself. I'm talking about the things the ai can choose to do which will change the score compared to a diffrent play through.


For example

Bioshock

Nvidia play throuhg - You walk around find a big daddy and you fight him . You get 50fps

Ati play through - You walk around find a big daddy and fight him , during the fight a second big daddy enters your area and while your fighting the first he becomes enraged also and attacks you. The card gets 40fps

Which card plays better ?

Right so now you need different canned benchmarks for ATI and NVidia.
 
Are you incapable of noting TRENDS in the graphs? It doesn't matter if card X got slightly better fps than card Y during time N, what matters is the overall trends.Smoke grenades, rockets, etc... everything you have stated only lasts for a couple of seconds, not an entire level. They account for all the peaks and valleys in the graphs. HOWEVER, a slight peak and/or valley doesn't change the overall trend. Playing through an entire level several times, taking roughly the same path and making the same decisions will give an average that is repeatable - try it out yourself. Play the same level in more or less the same way twice and compare your average FPS.

For example, take the Crysis apples-to-apples ( http://enthusiast.hardocp.com/image.html?image=MTIwMTIwNDUyODZqWk1RbU45c1hfM180X2wuZ2lm ). Is the 8800GTX always on top? No, of course not, the card in the lead varies at different points, but overall which card is on top a majority of the time? The 8800GTX - therefore the 8800GTX is better for Crysis. Also, note the peaks and valleys are generally mirrored between the two cards. When one card spikes, the other spikes too, which means that the two run throughs are much closer in what is being rendered than you are willing to admit.

But still, this is a 10 MINUTE run through of a level - various events that you mention will quickly average out and become insignificant over the time period. 20 seconds of marginally lowered FPS due to a smoke grenade over a 10 minute period will have a minimal effect, and it is highly probable that the other card will have experienced the same (if at a different point in time - it doesn't matter).

Well put. Sadly, unlikely to penetrate the fog. The same user said all the same things when the 2900XT review came out, the review which was later proved right many times over in forums across the net.
 
I'm wondering how much of a view/subscription spike this site gets when it reviews something 'big' like a high end GPU.

I'm also wondering if it would be worthwhile (or even possible) for developers to create better benchmarking tests that more closely simulate actual gameplay. I think the main issue has been answered already: There's no value in it for the devs, and plenty of potential pitfalls. Also, they could still prerender or optimize certain effects without our knowledge, and finally, it would be a ton of work and I'd rather see them fixing bugs. :D

Think about what it would take to run through a level in Crysis and have everything behave exactly the same. *shudder*

Unfortunately what this review says to me is that I won't be running at native res. in Crysis for a long time. (Yes, I realize that Crysis is the worst game in existence and looks just like Far Cry... :rolleyes: )
 
And now to spark the debate even further we have a review, with in game testing, with the 3870x2 outperforming the Ultra in many games. Comment from Zarden of DriverHeaven.net:

I was a little disappointed with Hardocp and their stance today on their front page, stating that apparently everyone else is using "canned" benchmarks and they are the only ones who aren't.

We haven't been using premade benchmark scripts for a very long time (we also dropped 3dmark in reviews before hardocp did), stuart uses real game play.


The review is here, I have to say this is by far the most informative review I have seen so far.

http://www.driverheaven.net/reviews/AMDX2Review/index.php

The biggest difference I notice is they are running a quad at 3.4ghz and they are using the newest "beta 5" drivers. In all fairness they do state there was a minimum FPS issue in three of the games they tested they asked AMD about it and it is an interesting response. Problem seems easily fixed with optimized drivers though. Good read there!
 
Right so now you need different canned benchmarks for ATI and NVidia.

Not at all . You need canned benchmarks that show the majority of a game.

Most likely you need 2-3 benchmarks per game as we all know that performance cna drasticly change from level to level.
 
i support [H] benchmark practice, thank you for the excellent review.
 
Kyle,

First off let me say I think the [H] way of testing is the way to go. But I do have a couple questions and suggestions that I thought of while reading this thread. I knew this was coming as we are now seeing big differences between benchmarks and actual gameplay.

1. You mention that the way you test is to play multiple levels and find the best settings. For a game like Crysis, you said you included the beginning and the later ice levels. If I recall correctly, isn't there a big difference in the graphical performance of those levels? If so, what does the review gameplay graph show? Is it a particular level or the entire testing session of all levels?

For a game like Crysis which might have two different areas with very different performance levels, either benching them both separately or just the most intense level might be the best. Either way, marking would be great so we know.


2. Maybe it's time for an article on canned vs. actual gameplay. I know you had an article in the past, but this would reintroduce everyone to your methodology (which you could hotlink to in replys) but would also show the world how it is becoming an issue. Seems like the perfect time to do so.

I would love to see "Crysis canned benchmark showed 38 fps on the 3870 X2 but each level ranged from a great 44 fps for the island portions to a measly 18 fps when we reached the final levels."


Thanks for the work, though to put these together!
 
Not at all . You need canned benchmarks that show the majority of a game.

Most likely you need 2-3 benchmarks per game as we all know that performance cna drasticly change from level to level.

The only reason [H] would need to show canned benchmarks would be to quieten down those that think there must be something wrong with the Hardware or config Kyle/Brent use.
The results [H] got speak for themselves.
Its even more work for them to include other benchmarks that dont show how it performs in games so I understand why they dont include them
 
And now to spark the debate even further we have a review, with in game testing, with the 3870x2 outperforming the Ultra in many games. Comment from Zarden of DriverHeaven.net:

I was a little disappointed with Hardocp and their stance today on their front page, stating that apparently everyone else is using "canned" benchmarks and they are the only ones who aren't.

We haven't been using premade benchmark scripts for a very long time (we also dropped 3dmark in reviews before hardocp did), stuart uses real game play.

The review is here, I have to say this is by far the most informative review I have seen so far.

http://www.driverheaven.net/reviews/AMDX2Review/index.php

The biggest difference I notice is they are running a quad at 3.4ghz and they are using the newest "beta 5" drivers. In all fairness they do state there was a minimum FPS issue in three of the games they tested they asked AMD about it and it is an interesting response. Problem seems easily fixed with optimized drivers though. Good read there!

I just read through the review, and it gives no indication as to what benching methods are actually used. Some specifics would be nice, but since the article told you what you wanted to hear all along, I'm sure you don't care.
 
Hmmmm, okay, well let's see why you think so...



Yes, you're right. What you don't seem to understand is that [H]'s method tests and reveals this problem, and the benchmark method does not. [H] tries EVERY resolution and setting before choosing a "best playable," in order to uncover exactly the issues you describe. The other sites don't--they pick "standard" resolutions and settings at the start and use them for everything. Just because [H] only shows the ones that worked the best doesn't mean they didn't do all the rest. That's how they chose which settings were best.



Like, umm, doing "gonzo world" benchmarks? Just read your statement again and see how hilarious it is. Let me rephrase it for you: "There is a LOT more to finding out the truth than collecting accurate information. You need to gather all the untrue information too."



Why? Is having immature drivers somehow a point in ATi's favor?

I NEVER said anything about "untrue information" Spin it how you want.... whatever. If there are massive differences between a single resolution or with a single setting on/off it either points to a bug in drivers or a seriously flawed game or piece of hardware. I like to see the results of different resolutions/settings. If you've tested every possible combination as you seem to imply... why not put them in the review?

Also... I really can't stand to play a game when it drops below 40fps. A lot of your "highest playable settings" seem to have a min fps below 30 and a lot of the time below 20. That is totally unacceptable in my book.

Or anything about ATI having immature drivers being in their favor. I was just trying to point out that ATI hasn't had near as much time to work on the 3xxx series drivers as Nvidia has had to work on the 8xxx series drivers.

In any case... I would like to see benchmark scores that use the SAME settings for both cards. Your "highest playable settings" is not always what the end user is going to need or want. For example... do you actually think that most people have huge monitors that can do super high resolutions?

I only have a 19" flat screen that does a max of 1280x1024 that I use to play games. My other monitor is a 19" flat screen that does 1440x900.

I don't need a bigger monitor at this point and the high resolution testing doesn't show me much of anything except that I could probably max out every other setting at 1280x1024 on either card.

I really would like to see benchmarks at lower resolutions.. not just the highest.
 
FRAPS is real-time, whereas timedemos are not real-time. A timedemo is a number of frames (say 3,840) that are rendered sequentially as fast as the system allows. Take the number of frames (3,840), the time it took to render those frames (say it took 90 seconds), and you have your 'frame rate' (42.6fps).

FRAPS doesn't record gameplay and replay it, it simply logs the actual frame rate at intervals, and these logs are used to build a graph.

Timedemos are real-time and exact copies of actual gameplay (those sequences happening during the record process) when you record them. Depending on the engine (most of the new games released in the last year) these timedemos include the full physics and capabilities of the game, which is what you want to capture and then replay from both a consistency and "actual" gameplay perspective.

The choice comes down to whether you want to replay this timedemo with the game engine and have it report the FPS data or with the game engine with FRAPS running and have it report the data with its associated overhead. As far as FRAPS capturing the gameplay, I did not word that right (unless you have a developers version), meant using the playback capabilities of the game engine with FRAPS running to capture the FPS.

The question is then, does Kyle and the gang record timedemos and then replay them with FRAPS running to generate their numbers or do they try to "simulate" the same action in a level by playing through it each time. If so, then how can you trust the results as the action will never be linear or exact each time. You cannot ensure consistency of the results using that methodology and therefore, the results are at best, an estimate of performance based on the actions of the user and game engine, not so much the video card depending on the variability of the action sequences.

If they record gameplay using the timedemo functionality and then playback this demo with FRAPS, then you capture consistent gameplay data with the only variable being the overhead incurred with FRAPS (which can vary from 2% to 7%). If they do this, then how is HardOCP really any different from the other websites that use FRAPS or timedemos?

What I think is important here is exactly what method does the [H] use when capturing performance data with FRAPS, is it timedemo replays or user(s) trying to simulate like gameplay in a particular level.
 
I don't see what's disappointing. Performance below a GTX, priced significantly below a GTX (in my country)? Good enough for me. Were people expecting an Ultra-SLI beater?

SLI and CF and dual-GPU are gimmicks in my opinion, until things start to scale "properly" and get proper support without, that means an end to Nvidia TWIMTPB titles that all the other sites test (seriously, who the hell plays F.E.A.R?), but at least the price of this two in one is less than two separates.


Car analogy:
To me, [H] tests is like taking it out on the road and going for a spin, down straights and around curves. Other sites tests to me are like propping it up on spinners and seeing how fast it can accelerate artificially with no wind or road conditions to think of, getting raw numbers that don't mean much unless you hook it up in a garage and run those tests yourself - how many of you guys use 4xAA 16xAF in every single recently released new title that demands a motherload of performance? How many of you skip AA and AF in exchange for better framerate and more enjoyable gameplay?

The [H] equivalent of "What settings do we need to use to keep it at a comfortable 30 fps average" is the car equivalent of "Is it a car, does it go fast, is it safe and fun". Never change, [H]!
 
I support [H]'s way of reviewing video cards, though I still think there can be improvements to such a way of doing it (maybe they are already?)

Taking the Dyno (canned) vs Track (real-world) analogy of how to test a video card and being part of of national competing NHRA pit crew (who does 7 second runs) I would like to further this analogy some more

When we get a new engine, or rebuild a current engine and have it dynoed we get x results, or course this does not translate to track results as we can all agree on. Then when said setup goes from test & tunning runs we get x results. Now each of them are slightly different, even when using electronics. We have to tweak settings and what have you (not exactly my responsibility btw) and run again. Still results vary, both internally (the engine and electronics) and externally (driver & track conditions (wind, traction, left or right lane))

Once we get the car where we want its time to do race passes... still this does not guarantee same results, still might not need more tweaking depend on the internal and external forces (could even be a new track at a different elevation, etc.)

What I am getting at here is..

is [H] taking various 3870x2 and 8800GTX cards? i.e. using 3 of the same brand for each and also using 3 more of 3 different brands and then combining the results through averages?

I would think the above would be more scientific and hold better results, just like for the drag team I am involved with, even the same engine with a new rebuild we give you different results, its just how things are.. it is quite possible that two cards of the same type from the same manufacture might have similar but different results (kinda of like cpu's right?)
 
I just read through the review, and it gives no indication as to what benching methods are actually used. Some specifics would be nice, but since the article told you what you wanted to hear all along, I'm sure you don't care.


Click the link that says discuss this in our forums at the end of the article :)


Here is the whole post, and there is some more information in that thread as well as in the review, each test is run three times per card and the results are averaged:



Thanks for the words of support today guys, much appreciated. Also be sure to check out the other reviews, we posted quite a few in the news today. I was a little disappointed with Hardocp and their stance today on their front page, stating that apparently everyone else is using "canned" benchmarks and they are the only ones who aren't.

We haven't been using premade benchmark scripts for a very long time (we also dropped 3dmark in reviews before hardocp did), stuart uses real game play. This is in fact how we noticed the minimum frame per second issue we noted (page 17 of the review), I don't think anyone else has even covered this in detail, take from that what you will! Not only that but we managed to get a statement direct from AMD explaining why. We have spent the last week working with AMD on the drivers, so I think its pretty fair to say we know what we are talking about.

That said, id rather not dwell on this, I hope our review has been informative, entertaining and educational.
 
I NEVER said anything about "untrue information" Spin it how you want.... whatever. If there are massive differences between a single resolution or with a single setting on/off it either points to a bug in drivers or a seriously flawed game or piece of hardware. I like to see the results of different resolutions/settings. If you've tested every possible combination as you seem to imply... why not put them in the review?

Also... I really can't stand to play a game when it drops below 40fps. A lot of your "highest playable settings" seem to have a min fps below 30 and a lot of the time below 20. That is totally unacceptable in my book.

Or anything about ATI having immature drivers being in their favor. I was just trying to point out that ATI hasn't had near as much time to work on the 3xxx series drivers as Nvidia has had to work on the 8xxx series drivers.

In any case... I would like to see benchmark scores that use the SAME settings for both cards. Your "highest playable settings" is not always what the end user is going to need or want. For example... do you actually think that most people have huge monitors that can do super high resolutions?

I only have a 19" flat screen that does a max of 1280x1024 that I use to play games. My other monitor is a 19" flat screen that does 1440x900.

I don't need a bigger monitor at this point and the high resolution testing doesn't show me much of anything except that I could probably max out every other setting at 1280x1024 on either card.

I really would like to see benchmarks at lower resolutions.. not just the highest.

Its no use, the Nvidia fanboys won't listen to anything you have to say. They want you to take any benchmark that favors Nvidia as the gospel truth and any benchmark that favors ATI as a complete lie
 
I honestly think [H] should do a 'real world vs canned benchmarks' article, just to show the differences between the two. It will go a long way to dispel the arguments currently being brought up.
 
The question is then, does Kyle and the gang record timedemos and then replay them with FRAPS running to generate their numbers or do they try to "simulate" the same action in a level by playing through it each time. If so, then how can you trust the results as the action will never be linear or exact each time. You cannot ensure consistency of the results using that methodology and therefore, the results are at best, an estimate of performance based on the actions of the user and game engine, not so much the video card depending on the variability of the action sequences.


You don't need each play through to be exactly the same. How do you choose what settings you play your games at? I assume you don't tweak them as you play...OH SHIT a bunch of barrels just exploded, I should turn down the res!. I assume you play the game for a bit then choose some settings and adjust after playing for a bit. Eventually you settle on ones that you feel play the best. This is what they do.

Many times with canned benchmarks I've had my settings at what was acceptable in the timedemo but find that the actual gameplay is much slower. FEAR is a good example of this, I had to use much lower settings than I'd picked after testing with the built in demo.
 
You don't make sense though.

I would agree with you had a way to get the same run through each time. However you don't.

I could understand using a time demo that you guys created to run through a section of the game. That way it would be the same each time. However since you don't your using two diffrent sets of data to come to a flawed conclusion.

If what I have said through the last few hundred posts does not make sense to you, then I would suggest that it would best for you to not waste your time reading our content if you cannot digest the meaning of it. We are more than a page of unanalyzed graphs that you will find elsewhere.

So we will agree to disagree. Your point is noted, you don't need to state it again. I understand your thoughts.
 
Click the link that says discuss this in our forums at the end of the article :)


Here is the whole post, and there is some more information in that thread as well as in the review, each test is run three times per card and the results are averaged:



Thanks for the words of support today guys, much appreciated. Also be sure to check out the other reviews, we posted quite a few in the news today. I was a little disappointed with Hardocp and their stance today on their front page, stating that apparently everyone else is using "canned" benchmarks and they are the only ones who aren't.

We haven't been using premade benchmark scripts for a very long time (we also dropped 3dmark in reviews before hardocp did), stuart uses real game play. This is in fact how we noticed the minimum frame per second issue we noted (page 17 of the review), I don't think anyone else has even covered this in detail, take from that what you will! Not only that but we managed to get a statement direct from AMD explaining why. We have spent the last week working with AMD on the drivers, so I think its pretty fair to say we know what we are talking about.

That said, id rather not dwell on this, I hope our review has been informative, entertaining and educational.

If thats not enough evidence, i don't know what is.
 

I don't know about you but when I get a game I don't just pick a resolution and AA/AF levels and play no matter what the FPS. I tweak the game so that I get a compromise of best IQ and best performance. That is what [H] shows. I am sorry you don't find that useful.
 
I don't know about you but when I get a game I don't just pick a resolution and AA/AF levels and play no matter what the FPS. I tweak the game so that I get a compromise of best IQ and best performance. That is what [H] shows. I am sorry you don't find that useful.

Good for you man. Do you want a pat on the back?
 
so can anybody give me a definet answer. is the new ATI card better then a 8800 ultra. a simple yes or no
 
Which of following cards I should now get:
8800 GTX 350€ ($515)
HD3870 X2 450€ ($660)

X2 would be 28% more expensive
 
so can anybody give me a definet answer. is the new ATI card better then a 8800 ultra. a simple yes or no

No.

However if price is an option the ATi card offers performance at a much lower cost, however it doesn't dethrone the Ultra at this time.

Which of following cards I should now get:
8800 GTX 350€ ($515)
HD3870 X2 450€ ($660)

X2 would be 28% more expensive

Need more info specifically resolution you typically game at.

Price / performance is a huge thing in my books so if I had to pick today i would go with the GTX in your situation, however the ATi offering is 449 and I went ahead and bought one with the Asus P5E today.
 
What I think is important here is exactly what method does the [H] use when capturing performance data with FRAPS, is it timedemo replays or user(s) trying to simulate like gameplay in a particular level.

But here is the rub, we DO NOT base our conclusions on the graphing you see of frames per second. We come to our conclusions based on real gameplay. We simply add the run through in as our proof to back up what we found while playing the game. The graphed run-through is simply to give our readers a look at what we are seeing while playing the game. Obviously it is also a tool to reinforce our findings as well, but graphed run-through is more for our readers than for us to form opinions with.

And we did discuss not doing those graphed run-throughs either, but I still think that is a bit more than the community could handle. Although I did think we would be beyond that five years later. :) The simple fact that we still think the "3dfx style" of benchmarking is simply a matter of economics. Sites just do not want to pay the costs of doing it right.
 
I was a little disappointed with Hardocp and their stance today on their front page, stating that apparently everyone else is using "canned" benchmarks and they are the only ones who aren't.

Where exactly did HardOCP make that statement on our front page? Please give me a link and the quote please.
 
Well I can't begin to elaborate how annoying having Gmail, Yahoo, and Hotmail all blocked was for me..but thats neither here or now.

What I'm interested in specifically is the FiringSquad review and its relation to your results here. The FiringSquad review certainly seems a lot more accurate than some of the others which have 2900XTs and 8800GTSs jumping all over the place.

Did you test any of the games in DX9 as well? This looks like it could be an issue with ATI's present DX10 driver performance, which would make a lot of difference for many people if they are at all like me (I still consider DX10 mostly worthless for the time being).

Still, this is the oddest place I've seen a GPU in ever before, of course I've only started really tracking them about a year and a half ago.
 
You don't need each play through to be exactly the same. How do you choose what settings you play your games at? I assume you don't tweak them as you play...OH SHIT a bunch of barrels just exploded, I should turn down the res!. I assume you play the game for a bit then choose some settings and adjust after playing for a bit. Eventually you settle on ones that you feel play the best. This is what they do.

Many times with canned benchmarks I've had my settings at what was acceptable in the timedemo but find that the actual gameplay is much slower. FEAR is a good example of this, I had to use much lower settings than I'd picked after testing with the built in demo.

Your example of FEAR is using the built-in benchmark. I am not even discussing that point. I am talking about using the timedemo functionality in games that support it to actually record gameplay and then using that recording to develop FPS data, now you can use the game engine in certain cases to generate it, you can use FRAPS to generate it, or you can write custom scripts utilizing GAWK (one example) to generate the same data. The outcome is a set of consistent results that utilize actual gameplay.

Utilizing this method allows you to replay actual gameplay to report consistent results at a variety of resolutions and quality settings. I play through a game to figure out what settings in general work the best also, at least from an overall viewpoint. However, when benchmarking, the results and test methodology need to be consistent.

If you play through a level with FRAPS running and generate those numbers, fine, but how do you do that consistently? You cannot, as the user or game engine will never generate the same results twice. Therefore, this testing methodology is not accurate and is nothing more than an estimation of performance within a single gameplay setting. If we hang our results on card A being 2FPS faster than card B with this method, then it is useless as the results will vary too much to generate meaningful numbers.

Who is to say the levels they play through are the right ones? Do they test multiplayer? I would just like to see the "actual" gameplay sequences they are using to report the numbers. Can I recreate that gameplay to see how well my system performs or do I continue to count on my seats of the pants feeling about what my system is doing or blindly trust the results. One good thing about built-in benchmarks is the ability to get 90% of the way there when it comes to setting up a system quickly or having fun comparing numbers against the systems of my friends, which usually results in a credit card being used a few minutes afterwards. ;)

Using timedemos, I can run a variety of settings with my system quickly to give me a fairly good idea of what settings work or not. Also, I can choose to record gameplay in a really difficult level to ensure my overall settings will be good throughout the game. It also gives me an opportunity to see how AA/AF affects the outcome, does it really make a big impact on visuals, if so, then I want to use it, if not, why take the framerate hit.
 
meaning no offense to many review sites, but I got sick of the graphs and charts and long ago.
[H] stood out to me because it wasnt just the absolute fastest the game could run, it was how fast it could run at acceptable quality and consistency.

Many times I looked at some chart, picked the card at the top and found out that I could run the game at some speed, but not play it.
 
Status
Not open for further replies.
Back
Top