What if I told you...

Status
Not open for further replies.

FrgMstr

Just Plain Mean
Staff member
2FA
Joined
May 18, 1997
Messages
58,702
What if I told you that I could prove that a timedemo in graphically intensive levels of Crysis when compared to the REAL WORLD run throughs scored about 15 - 25% framerates at the exact same settings? Yes, we are going to show you what we have known for a long time and of course to many [H] readers this will come as nothing new. Timedemos are not close to real world gameplay.
 
yes, let us resolve this issue once and for all so the editors of this site won't have to waste their time responding to the posts of slanderous trolls. there certainly will be detractors who question the real world gameplay method, but at least [H] will have laid its cards bare for all to peruse, critique, or dispel.
 
It'd be pretty irritating to know they've manipulated the performance from reality, especially if someone bought the game based on the FPS they expected to get.

What would really be devious business is if they altered the performance/FPS based on if the demo's run on an ATI vs. nVidia card ... then it'd really seem like a corrupt business act, if they not only falsely advertised the expected performance but influenced your purchase decision on a major computer component.

Then going back to the grand scheme of things... it is 'just' a game :p
 
Both have merit, while a timedemo won't show you what you can expect from the video card it will show you how it compares to other video cards running the same timedemo.

Though I personally prefer [H]'s method as over the years I've found it to be fairly accurate as to what I myself am able to get out of a graphics card when I buy it.


I wouldn't say one is better than the other, they both show performance. One just shows what you could / should get while the other shows how it compares to other cards in a way that doesn't show what you would be able to get performance wise from the card.

I don't see why the various review sites feel the need to say their way is better or whatever. Personally I wouldn't even make an article to address them, I'd just say their way is equally valid in a different way. Take the high road and don't stoop to their level.
 
Bring it. I'd love to see this settled, well at least in a few of our minds around [H] it already is, but still be good to see the naysayers at least quiet down a little bit.
 
Wonder what happens to the FPS if you do the whole rename .exe and then run canned vs. real world again... Would be interesting to see how many optimizations the fly-by has vs. the game :D
 
1. Both have merit, while a timedemo won't show you what you can expect from the video card it will show you how it compares to other video cards running the same timedemo.

2. Though I personally prefer [H]'s method as over the years I've found it to be fairly accurate as to what I myself am able to get out of a graphics card when I buy it.


3. I don't see why the various review sites feel the need to say their way is better or whatever. Personally I wouldn't even make an article to address them, I'd just say their way is equally valid in a different way. Take the high road and don't stoop to their level.

1. Yes a timedemo shows you what it scores in a timedemo, kinda like what a 3DMark score shows you.

2. Thanks, this exactly why we do it.

3. I could give a shit less about what other sites say, but HardOCP's superior methods get called into question and compared against sites showing other results. That is when I need to answer to our readers and let them know we are very much on track and showing them what is important and correct.
 
If a timedemo score 15-25% more with any card consistently, it won't matter because the end result would still be the same, if card A is faster than card B in timedome, it would also be faster in REAL WORLD than card B.


But I never said that......you are just making stuff up now. We are still working on the data. Crysis will be the game we use. COD4 was proving to have too many hurdles. Whodathunk recording your own demo in single player would be so hard? I wonder why they don't want you doing that?
 
I can't wait to show the motherfuckers are anandFUD about how wrong the yare.

Who the fuck are they to call HardOCP hardFUD anyways? shit pisses me off to no end. Atleast hardocp is still OG basement-building, game-loving, energy drinkin', integrity-wearin' mofos!

This will show them how "hacktheplanet" we are, REAL!
 
This will be very interesting to see. I'd like to see results in other games that have built in GPU tests (Lost Planet and FEAR come to mind), but I can understand how it could be a challenge to do so. I've been reading [H] for about three or four years now and this kind of an article will really help to see how your review style stacks up. I assume you'll be using the X2 and Ultra, but what other cards do you think you guys will test?
 
Get me front row tickets for this show. I hope all the thick skull's out there will finally get why your way of testing is very valuable to readers.

That isn't to say that timedemos are wrong. But, the reader needs to understand why your numbers are different. Numbers seem to be on the forefront of the **casual** reader's mind. If they could only understand that the numbers are only a reflection of the way the hardware was tested, and not necessarily the hardware's true ability. True ability being real-world gameplay.

I remember bying a TNT2 32MB back in 1998 or 1999. I often wondered why it didn't perform as well as what I read in a reviews that claimed it could do every game at 1600x1200 with no issue in all the newest games with full eye-candy.......Perhaps this argument could finally be put the rest.
 
If a timedemo score 15-25% more with any card consistently, it won't matter because the end result would still be the same, if card A is faster than card B in timedome, it would also be faster in REAL WORLD than card B.

It wouldn't show you the actual performance you would get while you are playing the game though, and that's what we want to do. We are evaluating the real-world experience delivered from each video card in games.
 
It doesn't matter, the argument is mainly that canned timedemos provide a controlled variable when you are conducting an evaluation of card versus card which makes for a fairer comparison.

We all know that timedemos score better than actual gameplay. We understand [H]ardOCP's position but most gamers want to know flat out which cards are defacto better performers. A best playble settings review like [H]ardOCP can be very subjective because what's best playable is up to the individual player. By knowing empircally via controlled experient with a canned timedemo, a gamer can make the best judgement of which card is superior or to extrapolate future performance or performance against other cards or their own system. Then they can themselves make the subjective judgement of what is best playable for themselves instead of the review telling them so.
 
WTF are you talking about spareflaire?
Thats like saying "OH I'll buy a Bugatti Veyron that can only go 0-60 in 7 seconds"

Don't advertise a god damn product that can't perform to what it's supposed to and then use timed demos to show NON real world performance.
 
It doesn't matter, the argument is mainly that canned timedemos provide a controlled variable when you are conducting an evaluation of card versus card which makes for a fairer comparison.

We all know that timedemos score better than actual gameplay. We understand [H]ardOCP's position but most gamers want to know flat out which cards are defacto better performers.

I really can't buy the unfair thing. I mean, reading the graphs of FPS/Time, I have yet to see one where the graphs did not have a nearly identical slope per given area. That tells me that these guys are looking at nearly the same scene. Now obviously that takes practice.

And if you think they are somehow cheating the numbers to make one company look better, ask yourself how and why. Instead of taking the time to graph the second card, they could just ad a constant to the cards score so it constantly is shown higher than the first card. Or, they could look at the sky instead of the the horizon...in which case the FPS/TIME graph would be WAY out of whack, obviously resulting inconclusive data. Besides that, why sell yourself out anyhow. You'll gain a few bucks today, lose a lot tomorrow.
 
WTF are you talking about spareflaire?
Thats like saying "OH I'll buy a Bugatti Veyron that can only go 0-60 in 7 seconds"

Don't advertise a god damn product that can't perform to what it's supposed to and then use timed demos to show NON real world performance.

Would you prefer comparing the abilities of the Veyron versus a superpowered Koenigsegg in the real world or on a controlled test track, running both under the same weather conditions on the same track with the same driver? Do you put more faith in the same professional driver driving both cars around the same track and getting a best time or do you put more faith in another driver driving both cars around real roads where conditions or driving may vary and then giving you a subjective review of what he felt both were best drivable at? If you watch Top Gear you may understand this analogy better. (you get both btw).

[H]ardOCP - you can avoid any of these issues by simply doing both kinds of test in your reviews. Show your real world performance and then show the canned timedemo. If what you say is true, you will be ultimately vindicated because in the same review under the same hardware, you can prove your point that real-world gaming is superior than a timedemo. You will also satisfy people in both camps and also provide a much better and comprehensive review than any other site...by being able to show the difference between real world and timedemo gameplay on every card you review. Give people the numbers instead of proposals or talk about what's better or not so they can make their own judgements.

As it stands, the conclusions are subjectively what the editor of the particular review feels was best playable for them with a graph showing the relative performance of each card in almost the same real-world gameplay but this is definetely not the same everytime (even though it is similar). Gamers relish hard numbers as do buyers who want to compare various cards, even if they are not in the same review (where a vga chart database like at Tom's, based on timedemos, comes in handy), it's just the nature of the beast.
 
It won't matter to me if you only prove that a timedemo is faster than REAL WORLD on any card consistently because the end result would still be the same. You should have asked:

In my experience, timedemo performance does not always translate linearly to real-world performance
 
In my experience, timedemo performance does not always translate linearly to real-world performance
I never said that it translates linearly or not, what I meant was, HardOCP needs to do better than just proving that a time demo is faster than REAL WORLD by 15%, 25% or even 200%, what they actually need to prove is that a time demo is X% faster on card A, Y% faster on card B and Z% slower on card C.

If they just write an article to show that the time demo is X% faster than REAL WORLD and they only use one card (card A) to prove this, I don't think that it would mean anything, at least not for me.
 
Sorry to be so blunt but hardocp show you how games will perfom on a narrow set of hardware with a narrow set of settings and use quite low averge frame rates to do so.

In this respect "canned" benchmarks are potentially more helpful since you can find the frame rate which suits you, and then look what settings were used to achieve that. This maybe 30fps, this maybe 60fps or this maybe closer to 100fps.

I think theres too much boasting about how good the [H] method is, but to me it seems like a very highly dedicated selection of the overall larger benchmarks that get done by other sites. Providing a subset of what others provide isn't especially helpful since graphics hardware doesn't always scale in a linear way when considering performance at different graphical levels.
 
I never said that it translates linearly or not, what I meant was, HardOCP needs to do better than just proving that a time demo is faster than REAL WORLD by 15%, 25% or even 200%, what they actually need to prove is that a time demo is X% faster on card A, Y% faster on card B and Z% slower on card C.

If they just write an article to show that the time demo is X% faster than REAL WORLD and they only use one card (card A) to prove this, I don't think that it would mean anything, at least not for me.

wouldn't it kind of be ridiculous for them to attempt this kind of comparison using only one card? i mean, if the test is only using one game with one card it would be a paragraph long at best. who would want to defend four years worth of methodical work by only the scenario described above?
 
Sorry to be so blunt but hardocp show you how games will perfom on a narrow set of hardware with a narrow set of settings and use quite low averge frame rates to do so.

:confused:

You mean they max out the IQ in games until the framerate is unbearable, then drop settings until they have a mix of the best quality/performance?

Oh right, just like anyone with a high end card/rig does with a demanding game/application?

I see. :rolleyes:

How narrow and limited. How dare they not show 100FPS AVG at the lowest IQ settings in their reviews.

I think theres too much boasting about how good the [H] method is, but to me it seems like a very highly dedicated selection of the overall larger benchmarks that get done by other sites. Providing a subset of what others provide isn't especially helpful since graphics hardware doesn't always scale in a linear way when considering performance at different graphical levels.

A subset of what others provide? Are you for real?

I have to ask...why on earth would you be buying a $450 card, to make your games look like complete ass, but have a high framerate? Don't get it. I guess I never will.
 
Sorry to be so blunt but hardocp show you how games will perfom on a narrow set of hardware with a narrow set of settings and use quite low averge frame rates to do so.

In this respect "canned" benchmarks are potentially more helpful since you can find the frame rate which suits you, and then look what settings were used to achieve that. This maybe 30fps, this maybe 60fps or this maybe closer to 100fps.

I think theres too much boasting about how good the [H] method is, but to me it seems like a very highly dedicated selection of the overall larger benchmarks that get done by other sites. Providing a subset of what others provide isn't especially helpful since graphics hardware doesn't always scale in a linear way when considering performance at different graphical levels.

how can you when most sites never indicate specifically what kind of settings they use outside of af and aa. even when they print aa, they don't tell you if they're using transparency aa or standard aa. to me, the issue between hardocp testing and the other high traffic sites (legit reviews, anandtech, tom's...) comes down to how thoroughly the authors of the articles explain how they achieved their results at the end of their charts. can you honestly say that about any other site other than [H]?
 
Yes you never said that but you asked

It won't matter to me if you only prove that a timedemo is faster than REAL WORLD on any card consistently because the end result would still be the same. You should have asked:

Trust me, I said exactly what I wanted to say. :) I am not putting all my cards on the table at once. ;)
 
I think theres too much boasting about how good the [H] method is, but to me it seems like a very highly dedicated selection of the overall larger benchmarks that get done by other sites. Providing a subset of what others provide isn't especially helpful since graphics hardware doesn't always scale in a linear way when considering performance at different graphical levels.

Thanks for giving me another opportunity to mention just how superior our methods are and that bascially everyone else running canned bencmarks is lying to you. :)
 
yep, sounds about right
i remember the same discussion back in the Quake 3 Arena times
with the crusher demo and all
FRAPS FTW !!!
 
1. That is the problem with [H] reviews, they can only test a few games with a few settings because it is too time consuming.

2. Let me give you an example from their latest review:They only provide the result with one setting in their review, if I have a 30" display, I would be more interested in running @2560x1600 with 2X ADSS AA than @1920x1200 with 8X ADSS AA and just by reading their review, I don't know which card would be better for me at this setting because almost every card behaves differently with different graphic settings, AA and resolutions. At least with canned benchmark, I know that the only difference is the card.

1. Yes, resources do come into the mix, but we have meetings all the time and discuss what games need to be dropped because they are simply no challenge for high end hardware. I can show you sims running on one at 2560 and 120fps and the other at 2560 and 160fps, but the fact is that is makes no difference.


2. First of all what you are griping about is not delivered by any site. To the second part, I think most of our astute readers can use a bit of their own brain power and look at the configurations and work out whether or not their own prefereces will work. I do know this, canned benchmarks tell you NEITHER about real world gameplay. To act like they do is a simple lie.
 
yep, sounds about right
i remember the same discussion back in the Quake 3 Arena times
with the crusher demo and all
FRAPS FTW !!!

That demo was awesome. The guy who built it works for AMD now. And the other guy works for Ziff Davis.
 
Yes you never said that but you asked

It won't matter to me if you only prove that a timedemo is faster than REAL WORLD on any card consistently because the end result would still be the same. You should have asked:

I think you are missing the point here buddy. It is in my personal experience that some demos are tweaked for certain vendors (i.e. Source Engine in time demos do better with ATi cards vs nVidia) even though in real world game play that may not be so. Also you have to look at it in the since that Single player mode vs online in a game are gonna put different kinds of stress on a PC and all of the key components (video card(s), RAM, and CPU). Even on my system, when I am playing single player mode in any given game, when I go to play online some times I may not be able to run the insane levels of AA/AF that I do in the SP.

Same theory applies when comparing time demos and "real world" game play, the time demo is like the Single player, and the "real world" is like the multi player. So lets say we were comparing video card "A" to video card "B" in a time demo. Said time demo benches card A at 80 FPS, and benches card B at 90 FPS. Then put in real world game play where nothing is PRE-RENDERED, say a 64 player BF2142 (or BF2) server card B gets 60 FPS and card A is getting 70 FPS, even though card B benched higher in the Time Demo.

Correct me if I'm wrong Kyle, but this is the theory and or facts behind this.:cool:
 
It just seems canned benchmarks would be as easy to manipulate as synthetic benchmarks. The timedemos might use the game engine but it is not the actual gameplay. I can easily see tweaks being made to drivers to capitalize on a small timedemo file that isn't part of the full game. In the past I bought graphics cards utilizing chips from vendors such as S3 and so on that showed impressive numbers running demos, but played the full game like, um, molasses being poured from a jar at subzero temperatures.(read: slow as hell)

What I gather here is that a large number of readers of hardware reviews want to be able to discern in ten seconds from looking at charts is which card is the "best". Cards perform differently depending on the game they are running since different games tax different resources. To me it's about what card is the best for the games I play
 
I never said that it translates linearly or not, what I meant was, HardOCP needs to do better than just proving that a time demo is faster than REAL WORLD by 15%, 25% or even 200%, what they actually need to prove is that a time demo is X% faster on card A, Y% faster on card B and Z% slower on card C.


In this respect "canned" benchmarks are potentially more helpful since you can find the frame rate which suits you, and then look what settings were used to achieve that. This maybe 30fps, this maybe 60fps or this maybe closer to 100fps.


Both of these statements have the same flaw. If you are looking for 60fps with certain settings and the system is being tested in a way that boosts it's performance by up to a quarter all of the sudden your 60fps becomes 45.

Worse yet, if you buy a card that you know will barely squeek by with 30 fps and you lose a quarter of that, you're stuck at 23. I don't know about you, but 23 for me is a little low. I doubt the [H] is out to prove that card A, B, or C is capable of a certain speed in a certain game, they do that already. What they're out to prove is that their system is the most accurate available. Someone's gotta be the best. And at least [H] is proving why their system is. The only argument I have seen that disproves this is that the [H]'s results are not 100% repeatable. But are we doing scientific experiments or playing freaking videogames? The last thing that I want out of my Team Fortress 2 is repeatablility (unless...y'know...I just nuked a spy, I could probably do that all day and not get bored :D)
 
1. I know that 100+ fps makes no difference already, for me 45+ avg fps is good enough. What [H] doesn't tell me is which card performs generally better or worse on which games, [H] only tells me that card A is better in game X at Y settings. [H] also doesn't tell me how these cards react with increase of resolution or increase of AA.

2. Yes it is not delivered by any single site, that's why I read multiple sites. :eek: From multiple sites, I know that Crysis performs better on a nVidia card, from multiple sites, I know that HD3870X2 doesn't take as much hit with a higher resolution as a 8800GTX and from multiple site, I know that nVidia cards are better with higher AA.


Lol read the comparison charts?:rolleyes:
 
What I'd like to see is what all this late model, high powered, whiz bang CPU/GPU HW brings to the table, or not, in Morrowind w/o the fog lifting View Distance mod. Can they deliver 45FPS, or better, at the entrance to Fort Monmouth? Or is the MW game engine just hopelessly broken, no matter the HW?
 
Both of these statements have the same flaw. If you are looking for 60fps with certain settings and the system is being tested in a way that boosts it's performance by up to a quarter all of the sudden your 60fps becomes 45.

Worse yet, if you buy a card that you know will barely squeek by with 30 fps and you lose a quarter of that, you're stuck at 23. I don't know about you, but 23 for me is a little low. I doubt the [H] is out to prove that card A, B, or C is capable of a certain speed in a certain game, they do that already. What they're out to prove is that their system is the most accurate available. Someone's gotta be the best. And at least [H] is proving why their system is. The only argument I have seen that disproves this is that the [H]'s results are not 100% repeatable. But are we doing scientific experiments or playing freaking videogames? The last thing that I want out of my Team Fortress 2 is repeatablility (unless...y'know...I just nuked a spy, I could probably do that all day and not get bored :D)
I agree that [H] review would be the most accurate method but it is not certainly the best, I certainly won't decide to buy a card just by reading a review from [H] due to lack of games and lack of resolutions/AA numbers.

If [H] could double the amount of games, provide numbers for at least 1680x1050, 1920x1200 and 2560x1600 with different AA settings, I won't go to another site anymore :D

I certainly don't agree that [H] should provide a time demo numbers with the same settings like some have suggested before. The only problem I see with [H] results is not the method but the choices of games and settings used for the review which sometimes doesn't reflect what people use/need.

Lol read the comparison charts?:rolleyes:
Yes, when I read the reviews from other sites, the actual numbers are not that important to me, I just look at which card is generally better in which games, which card has a bigger hit with higher resolution and which card has a bigger hit with higher AA. [H] review confirms the trend I saw on the other sites:

Crysis runs better on an nVidia card:
We also did an apples-to-apples test with the ATI Radeon HD 3870 using “High” Shaders Quality to compare directly with the GeForce 8800 GTX. Both video cards in the graph below are at the exact same settings.

ATI card suffers less performance hit with higher resolution:
The GeForce 8800 GTX also allowed us to play at 2560x1600, but only as long as AA was disabled at that resolution. Therefore at 2560x1600 the 8800 GTX and Radeon HD 3870 X2 are dead on in performance

ATI card gets a larger performance hit with higher AA:
At 1920x1200 or lower though, the GTX seems to allow a higher playable AA setting in Call of Duty 4.

So instead of giving numbers for just one particular setting and resolution to declare a winner, it would be better if they just ditch the fps vs time graph and replace it with min, avg, max fps graphs for different resolutions, AA and test more games.
 
I don't think this debate can be resolved. Both methods of testing have their pros and cons. That's why software developers use several methods of testing. Black box, white box, user, unit, regression, etc. Software is a complex thing, and that's one of the things being tested. The driver is generally the most important part of any test and it's software.

Timedemo is a white box test where the variables are known. It provides a consistent set of conditions, and is a good comparison of which card performs better than another under the same circumstances. It has a high level of consistency, but lower accuracy (as it relates to a "real world" experience).

Real world is a black box test, where the variables are not known. In any game the FPS will vary, depending on what's being rendered at a given point in time. So it's less consistent, but a more accurate measure of what we experience.

Neither method is actually better or worse, just different.

Think of it this way: Nvidia tweaks their drivers to get a higher timedemo score. Great, they've optimized their software. But we may still be disappointed with the real world results, due to increased variablility. Both can be true. They appear to be conflicting outcomes, but they're not.

My point is that both methods are valid for what they're meant to measure. I suspect most us look at several sets of benchmarks before buying a card, so I'm thankful to everyone that does the work that goes into them.

Anyway, just my two pennies (which don't make them fact)...
 
Ok so let me get this straight. The people who like time demos are basically saying they want to see 2 cards compared to one another using canned time demos so they can see like results in a controlled setting correct? Thus they can sompare which card is faster, in this controlled setting, and this does what exactly? Tells them which card can run a time demo faster and or which company optimized their drivers for said demo better? Ok so I go buy a $400-$600 video card and I base my decision on which card is better/faster/more worth my hard earned money based on one running said time demo better than the other card.

Hmmmm ok so then I get the card installled in my rig. I fire up some of those hot games I've been lusting to play on my new hardware and I find out that said game, isn't as fast on this hardware I purchased, as the review sites with the canned time demos showed.

Now I look at a review site that showed me some real world results, the same results I will see when I plug that hardware into my own rig, and I see pretty much WHAT I WAS SHOWN TO EXPECT and this is a bad thing??

I know I'm playing devil's advocate here but I hope my point is proven.

There is no reason in the world why said consumer would want/need to see old fashioned canned demos. No reason what so ever when I will never get said performance as was shown in that demo. So I ask you all, why do you care?
 
Status
Not open for further replies.
Back
Top