Firing Squid Wastes their Ink

[CaiNaM]

and the final say goes to Unreal's Tim Sweeney (game god):

"GD: Can you share some of the reactions you’ve had from leading game designers about Conroe?

Dan: Take it from Mr. Unreal himself, Tim Sweeney: “"Intel's new Core 2 Duo architecture brings gamers an entirely new level of CPU performance. Its breakthrough dual-core computing performance is ideal for Unreal Tournament 2007 and other Unreal Engine 3 games which exploit multiple threads to accelerate computationally-intensive animation, physics simulation, rendering and AI. We're looking forward to developing and playing our games on Intel's new Core 2 Duo CPUs. Core 2 Duo's performance is unrivaled!"”

enough said ) now..stay down horse!

http://www.gdhardware.com/interviews/intel/conroe/004.htm

sounds suspiciously similar to a commercial endorsement....

 

[HOCP4ME]

And with that comes the official close of this argument. We will now let it fade into computing history, and everyone will quickly forget about it, like they did with the R520 incident. It will be up to Kyle and the rest of the [H] crew what ideas, if any, they will use from this topic in their next review. But all that can be said has been said.

(Of course, I am not a mod, and I can't stop the debate, but I really think we should stop arguing, unless someone has a real question.)

 

[danman]

sounds suspiciously similar to a commercial endorsement....

You mean like the hardocp website?

 

[RAutrey]

sounds suspiciously similar to a commercial endorsement....

I don't know... He was one of the very few who was willing to say the P4 architecture was a piece of crap.

 

[bingo13]

Thank you for contributing to this topic.


Time constraints cannot have anything to do with it. The real-world tests that [H] does take many days to complete. Loading some games and running some timedemos takes a few hours (unless there's more to it that I'm missing here).

In fact. Kyle even posted low-res gaming results in the Conroe article several times. If he ran the benchmarks even when he wasn't going to publish them, he can also run the benchmarks with the intention of publishing them.

First off, it does not take days to run FRAPs demos and the amount of time it takes to run custom timedemos is just about the same. They are both one in the same during the playback session and only differ by the amount of data captured. You have FRAPs "benchmarks" that take seconds to run just as easily as you can have custom timedemos that take 20 minutes to run. Steve admitted he spent a couple of sleepless nights doing the Core 2 Duo tests so this whole subject matter about taking days is only true if you spent a few hours a day doing it. Also, I still find it ironic that the real world tests did not include any CPU limited games. As I stated in another post, I really could care less which method was used but do care about the logic in the statements compared to actual results in other reviews on this site. From my earlier post -

Really, what is the difference between a custom timedemo run at 16x12 8xAA/16xAF and a FRAPs recording at the same settings. The fact of the matter is the custom timedemo will have a .5% variability and FRAPs will have 3~7% on average. They know this and this is one of main reasons why you see different "setups" used in every review, they cannot accurately repeat the same results with their FRAPs capture between gpus, cpus, or motherboards. The whole "real world" benchmarking process is about as accurate as Kyle trying to measure the differences in velocity and volume of expelled gas generated each time his dog farts.

 

[bingo13]

But aren't SLI and Crossfire setups part of the real world to? Or did I just imagine that they existed? As I said at Anandtech, the article didn't cover the cpu. It covered a bottlenecked video card. Brent, what happened when you threw Crossfired X1900's at both the FX and C2D?? Come on, you know you did. Right?

They also had the same Intel and Asus 975X boards the other sites had for review. The same ones that would support a 7950GX2 also. The bigger question is why did they use a very immature P965 board for their testing?

 

[HOCP4ME]

First off, it does not take days to run FRAPs demos and the amount of time it takes to run custom timedemos is just about the same. They are both one in the same during the playback session and only differ by the amount of data captured. You have FRAPs "benchmarks" that take seconds to run just as easily as you can have custom timedemos that take 20 minutes to run. Steve admitted he spent a couple of sleepless nights doing the Core 2 Duo tests so this whole subject matter about taking days is only true if you spent a few hours a day doing it. Also, I still find it ironic that the real world tests did not include any CPU limited games. As I stated in another post, I really could care less which method was used but do care about the logic in the statements compared to actual results in other reviews on this site. From my earlier post

Well, when they do real-world tests, they try settings after setting to find out which one is most "playable". How do you think they discovered that lowering the shadow quality in Oblivion made the game playable? I doubt it was just a lucky first guess.

Whereas with a timedemo, you just click a button and it runs.

As for timedemos vs FRAPS, I think it had something to do with AI and physics. But the difference between the two really is negligible. What matters is high-res vs low-res, not timedemo vs FRAPS. And while any timedemo is technically synthetic, a high-res timedemo should be considered real-world IMO.

 

[HOCP4ME]

The bigger question is why did they use a very immature P965 board for their testing?

I think they said the 965 was more stable because it was designed "with Conroe in mind".

But I agree, they should've at least tried the 975 for the sake of using the GX2.

 

[CaiNaM]

I don't know... He was one of the very few who was willing to say the P4 architecture was a piece of crap.

oh i know.. and i don't necessarily disagree with him; there's no denying C2D is a vastly improved architecture and substantial step forward. it was just words/phrases he chose to use.

 

[CaiNaM]

The whole "real world" benchmarking process is about as accurate as Kyle trying to measure the differences in velocity and volume of expelled gas generated each time his dog farts.

didn't he take some fraps measurement of that as an example of the benefits of realworld testing?

 

[zumzum]

Well, when they do real-world tests, they try settings after setting to find out which one is most "playable". How do you think they discovered that lowering the shadow quality in Oblivion made the game playable? I doubt it was just a lucky first guess.

I agree, there are SO MANY combinations of different real world settings, it really takes lots of time to find just ONE setting to bench and call it real world and then draw conclusion from it.

 

[GezusK]

Originally Posted by HOCP4ME
Well, when they do real-world tests, they try settings after setting to find out which one is most "playable". How do you think they discovered that lowering the shadow quality in Oblivion made the game playable? I doubt it was just a lucky first guess.

Problem is, maybe I (or anyone) prefer shadows over tree complexity, or drawing distance, or AA or AF..."playable" is hardly a meaningful measurement. Its opinion.

The trouble I have with HardOCP's reviews...mainly the video card comparisons, is the data you have to wade through to figure out which card is faster. You can no longer glance at the fps number. You have to check to see if the resolution was the same...then if AA was enabled, and what level, then if AF was enabled....

Basically, just shifting a way from simple fps....

Am I suppose to be able to figure out image quality and game experience from that? If all setting were equal, then image quality should be the same (I know its not, but comparing that requires displaying screenshots pointing out differences). After that, the fps (low, high, average) is going to determine my game experience.

Maybe I'm not understanding how the reviews work. But you seem to use timedemos, so that what's on-screen is equal for all cards. Then you play with the resolution and other settings until you get similar fps. It doesn't seem to be much different...just harder to interpret.

oh well...I'm rambled on long enough...hopefully I did keep some coherent thought in that...

 

[justin82]

Screw it, its all like political parties.

Pick the one you wanna believe and go with that. I say live and let live, and if anyone doesnt believe that, take em out back and shoot the motherfucker. (borrowed from george carlin)

 

[Pkirk618]

i just hope that all this was worth it.

 

[b0bd0le]

so kyle & co, how much money have you made off this lame drama?

seriously it seems that hardocp is run buy a bunch of gossiping girls

i think hardocp is quickly digging it's own hole

maybe kyle is just too HARD for his own good

 

[HOCP4ME]

so kyle & co, how much money have you made off this lame drama?

seriously it seems that hardocp is run buy a bunch of gossiping girls

i think hardocp is quickly digging it's own hole

maybe kyle is just too HARD for his own good

As I said, every time things calm down someone comes in here with a trolling comment like that with no actual contribution to discussion.

Every time.

 

[seanmcd]

Can't we all just get along?

Any type of benchmarking results have to be taken with a grain of salt. I never read just one review when trying to decide which product to buy anyhow. I read as many as I can find. I like hardocp.com's reviews. The graphs can be hard to read at times, as it often looks like the three or four video cards compared basically perform the same. I definitely want to see 3dmark benchmark scores - as it's a benchmark i've used extensively. It doesn't hurt to show some timedemo scores either. Not only that, it takes very little effort run a few raw benchmarks and make a graph showing what went "faster". Most people would LIKE to see the scores anyways.

I think the "real world" benchmarks are needed though. I think if I paid more attention to them I probably wouldn't have gotten my last few upgrades. The "latest" card or cpu the last few years are barely faster than the version it's replacing. I would probably still be happy with my AGP 6800gt and single core 3200+ winchester with the games and resolution I play at...

 

[zumzum]

I think the "real world" benchmarks are needed though. I think if I paid more attention to them I probably wouldn't have gotten my last few upgrades. The "latest" card or cpu the last few years are barely faster than the version it's replacing. I would probably still be happy with my AGP 6800gt and single core 3200+ winchester with the games and resolution I play at...

Exactly. Too bad its only when Conroe comes out now that some people start to realise the importance of *real world* benchmarking...

 

[SidewinderX]

I have to admit, I agree with FS about your core2 gaming article. I really enjoy the real world you do with video cards, but CPUs are different. Your gaming article presents all the CPUs as essentially the same.... which may be 100% true now, but what about in the future? When I purchase a system, I buy it to last me at LEAST 3 years. If I had just read your review, I would think that I would get the exact same performance from an older chip as the new core2, and that would probably lead my to buy the older, cheaper chip. It may be fine for awhile, but in a few years, it may not be.

I know in your rebuttal you mentioned that the multimedia article showed some differences. Well, good for it. If I were just a strict gamer, maybe I didn't read that article. I understand that you need money from advertising, and the more pages you can put your articles on = more ads, but I think spliting up your review of the processor was, overall, very detrimental to the review.

What would have made the gaming review better for me? Well, keep the real world stuff. That definitivly shows that currently games could be GPU limited. But add some theroetical benchmarks that show that the processor has more juice that might be tapped into in the future.

By not doing that I think you are doing a disservice to your readers who come out of that review feeling that the core2 is noting more than a more expensive CPU that will not add anything to your gaming experince.

 

[zumzum]

How about we do a netburst CPU vs K8 *real world* gaming re-evaluation now with this [H] real world method, then compare to the old results of all the 640x480 tests that were done on AMD's CPU to see if its indeed a bad way to test CPU gaming power?

 

[Loadedaxe]

rock on....I'd like to see that!

 

[savantu]

How about we do a netburst CPU vs K8 *real world* gaming re-evaluation now with this [H] real world method, then compare to the old results of all the 640x480 tests that were done on AMD's CPU to see if its indeed a bad way to test CPU gaming power?

Why would you like to see a 3.2Ghz Prescott running neck in neck with the supposedly "far better" gaming CPU A64 3200+ ?

 

[Loadedaxe]

I think he meant the higher end A64 vs Conroes

 

[savantu]

I think he meant the higher end A64 vs the Conroes

You're thinking wrong.

 

[Loadedaxe]

Indeed I am....misread the post, still it would be interesting....

 

[zumzum]

Why would you like to see a 3.2Ghz Prescott running neck in neck with the supposedly "far better" gaming CPU A64 3200+ ?

I just want to see whether 640x480 tests are so useless as some people say.

 

[Sandman]

Just some random thoughts as I read the article:

but I find their usefulness waning every day as the desktop computer diversifies and advances.

I'm not sure what "diversifies" means. My comp today has the same general parts as my old pentium 2 300mhz machine, and they both perform(ed) the same tasks.

Firing Squad gives you pages of canned benchmarks...without any or very little analysis

That's fine with me, I usually skip to the graphs anyway. I do see the value in explaining your experience with hardware (as [H] does), but I also want to see the raw power of the device. I guess I'm saying that both methods are beneficial.

We were the first to bear the expense of real world gameplay for video card evaluations years ago.

Very true, and you guys have set a golden standard for video card reviews. The issue is that a CPU review is an entirely different beast. Running high-res games with it only shows what the GPU can do, it tells us nothing about the CPU limitations of the system. This is a limitation that we can easily hit with a CPU-bound game or a future video card upgrade. The info given in the gaming article isn't invalid or useless, it just confirms what common sense tells you. As such, if I were writing the article, that information would have been under the title "How Conroe affects your games today". I then would have put the system in a very CPU-bound scenario and showed readers what Conroe brings tomorrow.

Finally, as a long-time [H] reader, I have to ask: how well did it overclock??!? Am I crazy, or was the original [H] about going nuts with the hardware and taking it to the edge? Where's the keychain giveaway??

 

[Dosomo]

Very true, and you guys have set a golden standard for video card reviews. The issue is that a CPU review is an entirely different beast. Running high-res games with it only shows what the GPU can do, it tells us nothing about the CPU limitations of the system. This is a limitation that we can easily hit with a CPU-bound game or a future video card upgrade. The info given in the gaming article isn't invalid or useless, it just confirms what common sense tells you. As such, if I were writing the article, that information would have been under the title "How Conroe affects your games today". I then would have put the system in a very CPU-bound scenario and showed readers what Conroe brings tomorrow.

Finally, as a long-time [H] reader, I have to ask: how well did it overclock??!? Am I crazy, or was the original [H] about going nuts with the hardware and taking it to the edge? Where's the keychain giveaway??

You hit the nail on the head. The way the gaming portion of the Core 2 Duo review was done showed us ABSOLUTELY NOTHING. Anyone who knows anything about computer hardware already knew that if you make the GPU the bottleneck, then you will see zero gains between different CPU's. What an amazing revelation! The only way you would have seen a difference in performance would be if one of the CPU's were incredibly old and slow.

It was also idiotic to make sweeping statements during that "review" about other websites telling "lies" with their benchmarks. I guess this is because only [H] knows the correct way to benchmark hardware. Funny how they didn't figure out this "right way" until this review. Take a look at previous [H] CPU reviews. They were all performed with the "canned benchmarks" that [H] now looks down their noses at.

 

[Dawill]

I read both reviews, and many others and was fully capable of drawing my own conclusion. So you guys have different philosophies on reviews/benchmarking/whatever you want to call it. There is no need to get into a piss contest over it. Why can't both sites stand behind their own merits instead of mudslinging. You guys are turning into politicians. So what office are you running for next?

 

[omegadot]

I read this article, as I read most artiles posted or linked from [H], as well as bit-tech, rage3d and a few others. When I get to the bottom of all these articles I usually feel like I learned something new, or gained something from what I read. In this case all that gained was this funny annoyed feeling that I wasted my time. Stop bickering and constantly telling us how useful and wonderful your reviews are. We already friggin' know you feel that your style of reviewing is the most useful you can think of and that you disagree with canned benchmarks in most cases. If FS wants to waste their space and time, let them.

Last I checked this was a hardware site, not a blog, why is this article Featured? Hell I see plenty of articles here that should be featured over this little rant.

 

[jon_k]

Kyle.... you're really beginning to trash [H]ardOCP's reputation with stuff like this.

I certainly see the point of realworld benchmarks (if I didn't I wouldn't visit this site), but attacking another site like this - there is just no need for that. You guys are supposed to be professionals, but this article sure doesn't show that (that's not a slam, but an observation).

Does the NYTimes lash out at some independent website that calls their paper "biased and liberal"? No, they are professionals. They let their journalism speak for itself. You ought to do the same thing.

 

[bingo13]

Well, when they do real-world tests, they try settings after setting to find out which one is most "playable". How do you think they discovered that lowering the shadow quality in Oblivion made the game playable? I doubt it was just a lucky first guess.

They probably read the Oblivion guide like everyone else did.

 

[Bicarus]

When the initial slew of Conroe reviews hit the web, I was like many and started reading them to get an idea of how it did on current games, apps, etc.

One major thing I found that was quite dissapointing to me from the reviewers was that the majority of the reviews used pre-made systems (i.e. Reviewers did not setup the system from installation of hardware to software -instead they just ran the tests or was given them by intel or whomever). This alone made me dismiss alot of the reviewing sites out there for the simple fact that to me it was alot of marketing ho-ha. Granted there were a few who actually did get a conroe chip(s) get current mobo's that support it and rand their timedemos on it, but seriously I am not about timedemos for a review.

First off timedemos will at best you a glimps of what the hardware can do for that brief period of the in the game. They don't run the gamut of the entire game where loadtimes,full blown ai, etc. And like what others have stated before - the majority of reviewers out there just do not have the hardware expertiese to setup hardware and run it the bench's like a gamer or app user would. That would be like making a brand new ferrari and giving it to grandma who drives at 20 mph under the speedlimit all the time and asking her how it ran...

Realworld gaming has a large leeway on how the tests are done, but I beleive those reviewers out there and their years of experience can come up with a good baseline of "realworld" benchmark tests they can run. Canned reviews are at best like I said just a snapshot of how the game will play on the part of the game it is run, but leaves out alot of things that depending on hardware aside from gpu/cpu could make or break a consumers choice on if they will buy the new hardware or not.

 

[RedStarSQD]

"One major thing I found that was quite dissapointing to me from the reviewers was that the majority of the reviews used pre-made systems (..."

apart from the people who actually got conroes ..like FS... others made a point to state they were limited by intel and that they would not be able to make a real conclusion until they could get their hands on some for themselves. That said....conroe did indeed look promising.

And guess..what they were right! There was nothing wrong with the first reports.

 

[CaiNaM]

How about we do a netburst CPU vs K8 *real world* gaming re-evaluation now with this [H] real world method, then compare to the old results of all the 640x480 tests that were done on AMD's CPU to see if its indeed a bad way to test CPU gaming power?

wouldn't matter.. using a GeForce4 MX @ 2048x1536/4xaa/16af HOCP could show us that anything besides a 500mhz Pentium3 is a complete waste of money.

 

[lefthandedgoat]

Real world benchmarks?

Real world like those P4 HT benchmarks that you falsified... then banned a good number of your loyal and knowledgeable forum goers over?

 

[mrtsherman]

Well I figure I will write out my take on this whole debacle and hopefully somebody gleans some insight from it. From reading forty or fifty posts I have seen the following trends. I include my thoughts on each point.

1. Synthetic benchmarks let the user see how a processor compares to other processors. Real World Gaming fails to show this because other components become limiting.
You have to change your mindset to understand why this is actually a GOOD way of testing. Computers are a tool. Like a hammer or screwdriver, only way more complicated. You buy the tool that gets the job done. I'm not going to buy a hammer to put a screw in my wall. Now what [H] wants everybody to understand is that Real World Benchmarking properly indicates what you need to game. Their conclusion was that with the current state of the hardware industry, a Conroe chip is no better than the existing Athlon64. So that leads us to point number two:

2. Real World Gaming does not let me see how the hardware will perform in the future.
By removing the video card bottleneck the end user can see what might happen as new hardware is released. If I needed to buy a new system right now, what should I purchase? From the [H] it sounds like an athlon64 will do just as well as a Conroe. This could be wrong in only six months and represents a valid point. I am not positive how [H] should address it. [H] prides themselves on giving consumers data that is factual. Making conclusions based on hardware that does not even exist yet obviously falls outside these bounds, but is still something that needs to be considered. How should they handle this situation? My take is that they should look at trends in the video card industry to try and guess how the next generation of cards will remove specific limitations and how upcoming games may be placing new stresses on video cards. From there they can devise what a possible testing scenario of the future would be and provide "Future World Gaming" benchmarks. This would be even harder than Real World Gaming benchmarks, but I think it could be done.

My second thought on this is that predicting the future may be too hard. Who knows what power requirements, memory limitations, chip upgrade limitations, quad sli, multi gpu vid cards, etc will arrise in the future and negate upgradability. It is so much easier to deal with the technological world of now as opposed to the technological world of what if.

3. [H] was too childish in their response.
I would agree with this. The initial arcticle was too inflammatory and their reply to FS was equally inflammatory. Even after the first mistake they should have used a reply to FS as a way to properly editorialize on their benchmarking style. I think that trend #2 is really where all the fuss is centered, and if [H] can find a way to properly address it then they can remain up on their pedestal. Until then they do not have a satisfactory answer to lower resolution gaming benchmarks.

4. [H] should consider more user perspectives in their conclusions.
I would agree with this statement. Not everyone is running a high end Athlong64. What about those that are slightly behind the curve, a lot behind the curve, and cannot even see the curve? Perhaps some recommendations for these perspectives would help to alleviate the thought that [H] is not recommending Conroe. In reality they are just not recommending Conroe for someone who is already decked out with the latest technology.

 

[bubbagee]

Good points, similar to my own view at first, but I think maybe you are letting [H] off a little too easy. Not doing the "canned" benchmarks - which are your best and only bet for at least guessing at future performance - is just silly and wrongheaded. Everyone knows the low-res marks are reliable for removing the GPU from influencing a CPU test. Low-res has and will remain SOP! Doing low-res in no way implies that a CPU will achieve those numbers in hi-res, so it isn't misleading at all to show those numbers. The obvious answer is to do both kinds of tests and then everyone has ALL the data they need to make a decision. Which is pretty much what everyone has been doing. LOL I bet you [H] will be back to the old way sooner or later and that nobody else ever makes this mistake again. Sad to see [H] get trashed so. How do you recover from such a stupid blunder? Reading over the thread here and those at all those other "lying" hardware sites has shown me that their method has impressed no one but a few low-brow hard core [H] fans and the ever dwindling crowd of never say die AMD fans who just can't believe their baby could ever come in second. Tragic and funny as hell to watch. You think maybe Kyle and his crew are waking each day to a little more of this realization? I don't think they would trash thier rep for a few days of heavy traffic. They just lost it and now they are cornered.

 

[Mr. D.]

I am wondering what the guys of [H] think of all the replies in this threads. I had expected the replies to be a bit biased toward [H] because, after all, this is the [H] forum. But form what I have seen (the last two pages) this is not the case. Most of the replies state that both methods have their merits and that everyone should stand by his/her own method. Most people seem to be annoyed by the mudslinging. So am I by the way.

By the way, I do appreciate that they have left the thread open even though they have received some criticism.

 

[HOCP4ME]

Can't we all just get along?

That would be really nice.

I definitely want to see 3dmark benchmark scores - as it's a benchmark i've used extensively. It doesn't hurt to show some timedemo scores either. Not only that, it takes very little effort run a few raw benchmarks and make a graph showing what went "faster". Most people would LIKE to see the scores anyways.

That makes sense. Kyle actually did do low-res tests, he just didn't publish them. That proves that there was enough time to do both types of tests.

I think the "real world" benchmarks are needed though.

QFT

Too bad its only when Conroe comes out now that some people start to realise the importance of *real world* benchmarking...

That's because there was so much hype over Conroe that people forgot the difference between synthetic and real-world.

but what about in the future?

Great point.

Well, keep the real world stuff.[/qoute]

Yes.

But add some theroetical benchmarks that show that the processor has more juice that might be tapped into in the future.[/qoute]

Exactly.

I do see the value in explaining your experience with hardware (as [H] does), but I also want to see the raw power of the device. I guess I'm saying that both methods are beneficial.

QFT!

you guys have set a golden standard for video card reviews.

I agree; the video card reviews are awesome, and they are where real-world testing can really shine.

I then would have put the system in a very CPU-bound scenario and showed readers what Conroe brings tomorrow.[/qoute]

I would too. Future performance is important.

It was also idiotic to make sweeping statements during that "review" about other websites telling "lies" with their benchmarks.

but attacking another site like this - there is just no need for that.[/qoute]

Yes, the article would have been much better without the "other sites are liars" comment. Other benchmarks are beneficial, too.

Real World Gaming does not let me see how the hardware will perform in the future.

Very true. Real-world has it's place, but it's big disadvantage is that it can NOT give you ANY indication of future performance.

My second thought on this is that predicting the future may be too hard.

It is hard, but synthetic benchmarks at least make it easier than real-world testing, which gives virutally no indication of the future whatsoever.

4. [H] should consider more user perspectives in their conclusions.

YES!!!

Not doing the "canned" benchmarks - which are your best and only bet for at least guessing at future performance - is just silly and wrongheaded.

Well, it's not silly if you aren't trying to measure future performance, but a good review needs to take a look at future performance.

Most people seem to be annoyed by the mudslinging. So am I by the way.

So am I. It would be great for the arguing to stop..

The obvious answer is to do both kinds of tests and then everyone has ALL the data they need to make a decision.

The exact point that just about poster in the thread, including I, has been trying to make (and have already made, IMO).

Anyone else?