• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

[H]ardOCP and Force Unleashed testing methods

P

POPEGOLD

[H]ard|Gawd
Joined
Dec 18, 2005
Messages
1,117
http://www.youtube.com/watch?v=4mBcS8vQL2Y

No sure if Star Wars : Force Unleashed is coming out on the PC.... but this game has some highly intergrated A.I and Physics built it.

If you Watch the video... the game is never the same twice. The environment and entities all can behave totally differerent each and every time u play a scene

I wonder how this will effect REAL IN GAME testing.

lets say in pass 1... you throw a crate at a storm trooper and it hits him and he falls over and dies.

in pass 2, he decides to duck and it hits a propane tank and explodes...

this can make it very hard to make the test even for different hardware.
 
this is the problem with most modern games.

However Hardocp has some magic mojo that lets them run the game the exact same way each time . :(
 
gljvd said:
this is the problem with most modern games.

However Hardocp has some magic mojo that lets them run the game the exact same way each time . :(
Click to expand...


i will need it explained to me how this game will test evenly from card to card...when the level wont play the same on each pass.
 
i guess if u run the same scene 5 times.. for each card and take an average

i guess that would be fair
 
POPEGOLD said:
i guess if u run the same scene 5 times.. for each card and take an average

i guess that would be fair
Click to expand...
This is what I was contemplating. Even without a special method of replaying the level exactly the same way twice, you can always take averages and they should still tell you very well how a given hardware will perform in actual game play.
 
APOLLO said:
This is what I was contemplating. Even without a special method of replaying the level exactly the same way twice, you can always take averages and they should still tell you very well how a given hardware will perform in actual game play.
Click to expand...


then u run into the problem...that one card can render scenes with alot of fluid faster

but another card can render certain shaders heavily used on a level better

ltos of variables
 
POPEGOLD said:
then u run into the problem...that one card can render scenes with alot of fluid faster

but another card can render certain shaders heavily used on a level better

ltos of variables
Click to expand...
Certainly, but different architectural designs will always result in different gameplay even if you make the testing environment as sterile as possible. That's why there are reviews, in order to highlight the architectural differences to prospective buyers of the hardware concerned.
 
umm could you not just play the same level like 50 times and get an average. Is that not how stats are done in real life?
 
Table21 said:
umm could you not just play the same level like 50 times and get an average. Is that not how stats are done in real life?
Click to expand...

Or you could just test the same level once, as even over the course of one level things will average out. The OP's theoretical scenario doesn't make any difference, and is constantly being brought up. What the people who keep bringing this up don't seem to understand is that it doesn't matter in the least. Take a game, any game, and play the same level in roughly the same manner twice while recording your FPS. Over the course of the level any differences average out over the course of the level. A 5-10 second explosion that occurs on one test but not the second will have an extremely minimal impact over the course of a 5-10 minute run. There is always a margin of error, even with canned benchmarks, and there always will be. Just deal with it and accept it.
 
I'm really looking forward to this game. We haven't had a good Star Wars game in a number of years now. The last good one was Republic Commando which I really enjoyed.
 
gljvd said:
this is the problem with most modern games.

However Hardocp has some magic mojo that lets them run the game the exact same way each time . :(
Click to expand...

Why do you feel the need to be a sarcastic clown?

Do some research and you'll see how you can do it.

Fact is that run to run, slightly different events on screen do not affect the framerate enough to alter results by much. This is especially true when you do MULTIPLE RUNS and average results together. This way it's entirely representative of the potential gameplay experience.
 
kllrnohj said:
Take a game, any game, and play the same level in roughly the same manner twice while recording your FPS. Over the course of the level any differences average out over the course of the level. A 5-10 second explosion that occurs on one test but not the second will have an extremely minimal impact over the course of a 5-10 minute run. There is always a margin of error, even with canned benchmarks, and there always will be. Just deal with it and accept it.
Click to expand...

Sr7 said:
Fact is that run to run, slightly different events on screen do not affect the framerate enough to alter results by much. This is especially true when you do MULTIPLE RUNS and average results together. This way it's entirely representative of the potential gameplay experience.
Click to expand...
Totally agreed with both these posts. I wish this pseudo-controversy can finally be laid to rest.
 
Well, this has nothing much to do with Force Unleashed but it is a good example of how gameplay can vary (and drastically). The answer is so simple, I don't understand why [H]ardOCP is so stubborn about it. I've said before:
[H]ardOCP - you can avoid any of these issues by simply doing both kinds of test in your reviews. Show your real world performance and then show the canned timedemo. If what you say is true, you will be ultimately vindicated because in the same review under the same hardware, you can prove your point that real-world gaming is superior than a timedemo. You will also satisfy people in both camps and also provide a much better and comprehensive review than any other site...by being able to show the difference between real world and timedemo gameplay on every card you review. Give people the numbers instead of proposals or talk about what's better or not so they can make their own judgements.
Click to expand...

Is it so hard to give people the best playable settings in real world gameplay type information WHILE also providing the evidence from a canned benchmark so people can interpret and extrapolate what they want from seeing both? It satisfies everybody. For a game like Force Unleashed (for similar games that will actually be released on PC), the only fair method would be to do something already suggested, repeat the same test about 10-20 times over and get an average as in real life statistical testing and provide a standard deviation (as also suggested by another thread). This method requires a lot more work then taking a timedemo and clicking on an exe or load demo...
 
1st. It's not coming to PC, so no go there.
2nd. kllrnohj says it pretty accurately, there is a margin of error in any type of testing, whether it's canned or real gameplay, which is why some websites will do a benchmark 3 or 5 times and take the averages of them. What I don't understand is how people have such a problem with [H], seriously, if you don't like it, move on, don't read it, nobody has a gun to your head. And you guys wonder why the writers/managers/owners like Kyle get a little annoyed, but I'm guessing it's because EVERY review they write they have people asking the same questions that have been answered before. Multiple times. If you don't like the answer, don't agree with it, well then that's when you are told you can kindly leave, because the answer isn't going to change.

I for one come here to read ALL the reviews because they're done thoroughly and done well. Motherboards, power supplies, video card, are all done with great attention to detail and cover almost all the bases. And I really do like the video card reviews, of course they're not 100% perfect, but they're a far cry from most of the half-assed video card reviews out there. I'm sure many were happy that [H] was able to tell everyone what a crappy video card the 2900XT was, saved many people a lot of money. It's the people who decided to trust the other half-assed video card reviews that had to go out and buy a GTX later because they weren't satisfied with their loud anti-aliasing inadequate power hog of a video card. That's the facts.

Or do you still disagree glvjd?
 
Dan_D said:
I'm really looking forward to this game. We haven't had a good Star Wars game in a number of years now. The last good one was Republic Commando which I really enjoyed.
Click to expand...

There haven't been very many Star Wars games in the past few years altogether. I did like Empire at War though. It wasn't an amazing game because of the fairly poor land battles, but the space battles were awesome.

As for the debate about the variance in performance since levels don't play out the same way every time, it's true that because of advancements in AI you won't see the exact same behaviour each time. However, the levels themselves are the same, as are the positions of the characters in each one. Although their behaviour may vary, the scenes generally take place under the same circumstances so performance doesn't vary a whole lot and the average FPS one would get in a certain area will stay more or less the same.
 
Zero82z said:
There haven't been very many Star Wars games in the past few years altogether. I did like Empire at War though. It wasn't an amazing game because of the fairly poor land battles, but the space battles were awesome.
Click to expand...

Agreed. I too enjoyed the space battles, but hated the land combat.
 
god damn PC games are getting boring with all the cool ones going over to console

god i want too kick an xbox in the balls right now
 
gljvd said:
this is the problem with most modern games.

However Hardocp has some magic mojo that lets them run the game the exact same way each time . :(
Click to expand...

Problem ? It's not a problem at all, it's actually a step in the right direction.

To the OP:

As for testing it, if it was a PC game, which it seems it's not, the real-world gameplay methodology would remain intact and nothing would change, since this game is no different than many others already out i.e. very dynamic.
 
gljvd said:
this is the problem with most modern games.

However Hardocp has some magic mojo that lets them run the game the exact same way each time . :(
Click to expand...

Even now after having our process explained to you multiple times you refuse to listen to reason and spout what is tantamount to a lie above. I am tired of not listening to answers. Your time here is done.
 
sorry if i got someone zapped... I was looking at a specific game in FORCE UNLEASHED that looks like its numbers can vary from run to run....however since it doesnt look like it is going to be released on the PC..its a moot point.

That said..even if games really change from test run to test run... i would guess a 5-pass average for all GPU's or CPU's involved with be a solid solution.

That would hold the most gravity in my purchasing choices and a reason i applaud [H].
 
POPEGOLD said:
http://www.youtube.com/watch?v=4mBcS8vQL2Y

No sure if Star Wars : Force Unleashed is coming out on the PC.... but this game has some highly intergrated A.I and Physics built it.

If you Watch the video... the game is never the same twice. The environment and entities all can behave totally differerent each and every time u play a scene

I wonder how this will effect REAL IN GAME testing.

lets say in pass 1... you throw a crate at a storm trooper and it hits him and he falls over and dies.

in pass 2, he decides to duck and it hits a propane tank and explodes...

this can make it very hard to make the test even for different hardware.
Click to expand...

To answer your question it would have little impact on our real world testing since we derive our highest playable settings from playing through what is usually the entire game.
 
Spare-Flair said:
The answer is so simple, I don't understand why [H]ardOCP is so stubborn about it. I've said before:
Click to expand...

Your point is noted and was noted before when you posted this the first time. Thanks for your insight. You have a great future in the hardware review site management business.
 
POPEGOLD said:
That said..even if games really change from test run to test run... i would guess a 5-pass average for all GPU's or CPU's involved with be a solid solution.
Click to expand...


From what you have said here, I am going to guess that you really don't understand our precess yet either.

The run through that you see graphed and that data is pulled from is shown for reader benefit only. That run through gives you data that "proves" our conclusions. It is nothing more than a graphical representation of what we experienced playing the game. The resolution, AA/AF, and eye candy in-game settings are determined through real gameplay of the game. We DO NOT determine our highest playable settings by using a run through.
 
Spare-Flair said:
Well, this has nothing much to do with Force Unleashed but it is a good example of how gameplay can vary (and drastically). The answer is so simple, I don't understand why [H]ardOCP is so stubborn about it. I've said before:


Is it so hard to give people the best playable settings in real world gameplay type information WHILE also providing the evidence from a canned benchmark so people can interpret and extrapolate what they want from seeing both? It satisfies everybody. For a game like Force Unleashed (for similar games that will actually be released on PC), the only fair method would be to do something already suggested, repeat the same test about 10-20 times over and get an average as in real life statistical testing and provide a standard deviation (as also suggested by another thread). This method requires a lot more work then taking a timedemo and clicking on an exe or load demo...
Click to expand...
[H] doesn't need to do both canned benchmark and REAL WORLD benchmark to show that there is a difference between them because we already know that there is a difference just by compairing [H] numbers with other sites.

What [H] needs to do is show how well the GPU scales with different settings(AA, resolution and game) and from there they can explain why their numbers look differently from others but actually the numbers from [H]'s X2 review is very consistent with other sites in term of GPU scaling with AA and resolution.

I've talked about this before here:
http://www.hardforum.com/showthread.php?t=1269634&page=4
 
AA is something i look at... but its not a focus of mines. I think its a little overated

IN fast paced FPS games....at 1650x1080 ... everything on high... does one actually notice AA.

On something like the coming STARCRAFT 3... i could really see AA being more important....but thats just my opnion.

I think gamers who are mostly FPS players with big monitors would sacrifice AA for sheer speed... Would be nice to poll that.
 
POPEGOLD said:
AA is something i look at... but its not a focus of mines. I think its a little overated

IN fast paced FPS games....at 1650x1080 ... everything on high... does one actually notice AA.

On something like the coming STARCRAFT 3... i could really see AA being more important....but thats just my opnion.

I think reaser who are mostly FPS players with big monitors would sacrifice AA for sheer speed... Would be nice to poll that.
Click to expand...

That is the problem with [H] review, they only provide numbers for one specific setting so people will have a hard time figuring out how well the card will perform with different settings like a higher resolution without AA or a lower resolution with insane AA and maxed out eye candy.
 
alg7_munif said:
That is the problem with [H] review, they only provide numbers for one specific setting so people will have a hard time figuring out how well the card will perform with different settings like a higher resolution without AA or a lower resolution with insane AA and maxed out eye candy.
Click to expand...

For a guy like me who doesnt use AA... i have to guage how a card would run at setting i run.... but i still have a good idea.

and really what it boils downs too... the Radeon x3870X2 gets the job done by Kyles number or by CANNED ANAND scores...


---------------------

I reserve the rights to "CANNED ANAND" as a trademark of 2008 POPE productions Inc. LLC lmtd ,
 
POPEGOLD said:
For a guy like me who doesnt use AA... i have to guage how a card would run at setting i run.... but i still have a good idea.

and really what it boils downs too... the Radeon x3870X2 gets the job done by Kyles number or by CANNED ANAND scores...

I reserve the rights to "CANNED ANAND" as a trademark of 2008 POPE productions Inc. LLC lmtd ,
Click to expand...

I think Brent Justice wrote a good review where he also pointed out what you said: Both cards get the job done.

I bought the 3870X2 after reading [H]'s review (and probably most reviews that were out then). All reviews told me that it would give me great performance in current games.

Driverheaven also used the real gameplay as method:

Good Benchmarking Practice:
Where possible, each benchmark was performed three times and the median result for each resolution/setting is shown in the tables that will follow. All applications had their latest patches applied and all hardware features the latest BIOS/Firmware. We use real game play to determine our results.
Click to expand...
http://www.driverheaven.net/reviews/AMDX2Review/testsystem.php

But, their testbed was different.

I ran the same tests as [H] in Call of duty, but with different testbed (os, machine, drivers etc). Ran through most of the game (not for test alone, but to play it again with more eyecandy) and I got totally different conclution: 1920x1200 4xAA 16xAF with higher average FPS.
http://www.hardforum.com/showpost.php?p=1032015747&postcount=66

None of the tests are accurate if compared to my setup, but they can give indication of what to expect. Canned benchmark shows the rendering performance in a certain scenario, while real time gaming as done with [H] shows a subjective experience with it (and the numbers are hardly accurate). The [H]'s review are still better in many ways, since its more like a user review and therefore more personal.

I find that the different reviews supplement each other and don't want to exclude neither "canned tests" or "real time gaming".

Sorry to say that Kyle, but personally I don't find any method as "the ultimate" most accurate one. [H]'s review did help me in my purchase though and that mattered most for me (and I'm a happy buyer). :D
 
Tamlin_WSGF said:
I think Brent Justice wrote a good review where he also pointed out what you said: Both cards get the job done.

I bought the 3870X2 after reading [H]'s review (and probably most reviews that were out then). All reviews told me that it would give me great performance in current games.

Driverheaven also used the real gameplay as method:


http://www.driverheaven.net/reviews/AMDX2Review/testsystem.php

But, their testbed was different.

I ran the same tests as [H] in Call of duty, but with different testbed (os, machine, drivers etc). Ran through most of the game (not for test alone, but to play it again with more eyecandy) and I got totally different conclution: 1920x1200 4xAA 16xAF with higher average FPS.
http://www.hardforum.com/showpost.php?p=1032015747&postcount=66

None of the tests are accurate if compared to my setup, but they can give indication of what to expect. Canned benchmark shows the rendering performance in a certain scenario, while real time gaming as done with [H] shows a subjective experience with it (and the numbers are hardly accurate). The [H]'s review are still better in many ways, since its more like a user review and therefore more personal.

I find that the different reviews supplement each other and don't want to exclude neither "canned tests" or "real time gaming".

Sorry to say that Kyle, but personally I don't find any method as "the ultimate" most accurate one. [H]'s review did help me in my purchase though and that mattered most for me (and I'm a happy buyer). :D
Click to expand...

I guess KYLES scores should have the same disclaimer as fat loss drugs

*results are atypical and may not reflect your actual scores on your particular PC setup with your particular settings. :D

but dont get me wrong ..as i have stated if i had to only look at one site for a review on something i wanted to buy.... [H] is it

HEY KYLE WHERE CAN I GET A [H] case badge?
 
I think everyone needs to read the Skull Trail Review on here, BOTH Canned and Play benchmarks were posted and the value of both really shines. Go, read the review and you will see the benefits of having both.
 
POPEGOLD said:
I guess KYLES scores should have the same disclaimer as fat loss drugs

*results are atypical and may not reflect your actual scores on your particular PC setup with your particular settings. :D

but dont get me wrong ..as i have stated if i had to only look at one site for a review on something i wanted to buy.... [H] is it

HEY KYLE WHERE CAN I GET A [H] case badge?
Click to expand...

The scores here are the closest to what you will actually get, and unlike 99% of the other sites out there, they actually tell you in detail how they arrived at their scores. The disclaimer you jokingly stated SHOULD read "Results are typical and will reflect extremely well what you will get when you play the game".

@TheBluePill: The canned benchmarks didn't show crap other than that by spending several thousand on a rig you can get great frame rates at 640x480 and 800x600, and the real-world just further showed that the CPU largely doesn't matter at higher resolutions - stuff I hope almost everyone here already knew.
 
TheBluePill said:
I think everyone needs to read the Skull Trail Review on here, BOTH Canned and Play benchmarks were posted and the value of both really shines. Go, read the review and you will see the benefits of having both.
Click to expand...

LOL, good one. I'm not exactly sure what you're saying here... yes the canned benchmarks really shine... for testing how multi-threaded a game is at lower resolutions.... in the real-world gameplay section the $3000 skulltrail rig came off looking pretty crappy for the money spent. If that's what you're saying, great, but if you are implying that 24 extra FPS in Lost planet at 800x600 is somehow useful other than showing that that game is multi-threaded, then I've completely lost your point.
 
Citizen86 said:
LOL, good one. I'm not exactly sure what you're saying here... yes the canned benchmarks really shine... for testing how multi-threaded a game is at lower resolutions.... in the real-world gameplay section the $3000 skulltrail rig came off looking pretty crappy for the money spent. If that's what you're saying, great, but if you are implying that 24 extra FPS in Lost planet at 800x600 is somehow useful other than showing that that game is multi-threaded, then I've completely lost your point.
Click to expand...

you know 800x600 is horrible.... i was playin this new MMO and it defaults to 800x600...it looked like atari 2600

at 1650x1080 it looks great

800x600 means nothing at all to me it should be banned
 
POPEGOLD said:
I guess KYLES scores should have the same disclaimer as fat loss drugs

*results are atypical and may not reflect your actual scores on your particular PC setup with your particular settings. :D

but dont get me wrong ..as i have stated if i had to only look at one site for a review on something i wanted to buy.... [H] is it

HEY KYLE WHERE CAN I GET A [H] case badge?
Click to expand...

LOL! :D

Joke aside, there are some truth in it. The reviews only reflect what they see under their chosen conditions. A simple driver change can give different results. I think the biggest point in reading [H]'s review isn't about the numbers themselves, but the experience they have with THEIR setup. :)

Conditions change if gameplay changes and so does the FPS. I don't get hung up too much about those numbers, though it was nice to see it performed well in numbers too. Its not so much a Nvidia vs. ATI for me, but more if the 3870X2 would serve my needs and it does. I'm happy!
 
You must log in or register to reply here.
Back
Top