Firing Squid Wastes their Ink

[aldamon]

Amen. Put into better words than I thought of. This is word for word what I've seen happen to HardOCP, except I would say Tech Report has surpassed Anandtech.

I go to TR first as well. My proxy at work blocks enthusiast.hardocp.com.

 

[bubbagee]

For my money your testing method DOES test one aspect of real world performance - meaning me playing a game today will probably not see a big difference with Conroe, if any difference at all. I was a bit surprised to see your results, after seeing all those pre-release benchmarks showing such a big lead for Conroe. So you performed a service for me. I have been out of the performance game for a while and was unaware we had reached a GPU ceiling. Now I know. Thanks.

HOWEVER, there is a problem with your article. The problem is one of tone and emphasis - not of factual error. Here is how I think your article should have gone:

"Well, yes, Conroe is a killer and now AMD lags everywhere. Conratulations to Intel. The sleeping giant is awake and mad as hell. But before you dump your Athlon look at these benchmarks. . . Basically we have maxed out current GPU's folks. This is NOT a fault of INTEL who have done all they can and made a superior CPU, which WILL be superior in gaming should GPU's catch up anytime soon. Don't get me wrong. Conroe is king, period. I only want to make a big point of this for my readers because the pre-release marks released elewhere might lead you to believe getting a Conroe today will get you recent AMD owners a bunch more frames. This is simply not true and I think others should have done a better job of pointing this out!"

This article would have made your point and it wouldn't have stirred a hornets nest. But this is how I read the actual article:

"Ah ha! Those syntehtic benchmarks and time-demos are simplistic and misleading. The Conroe ISN'T any better in 3d games, in a (big empahis here) real-world situation. Oh yes, it is because we have maxed out the GPU and Conroe may be better later. But right now, if you have a beffy AMD system, the Conroe is a waste."

Here's the subtext I hear in that " I'm pissed that everybody has read those pre-release reviews and is so convinced of Conroe. I'll make another article that, like the others doeasn't actually lie, but which will serve as a counter weight. For this I'll need to pull out a little polemical juice"

And there's the rub. As a reviwer you are counted upon to walk a strict line of impartiality and to NOT play any games. You are, or should be, Consumer Reports, not Bill O'Reilly. That is if you want serious readers to trust you.

And there IS a problem with your use of the term 'real-world' because that umbrella covers so much more than situations such as GPU maxout, which must be the LEAST important real-world fact about a CPU, no? Considering how quickly GPUs can and do catch up. Yes, you made a real-world observation but not one that warrents so much bluster. That SCSI is potentially much faster than IDE but never really is - on a typical desktop, and hasn't been or ever will be - well that's also a real-world observation. Is it really like Conroe/AMD in 3d? No. SCSI on a desktop is classic 'real-world' and your's isn't. You attempt to piggyback on classic situations by using the un-qualified term.

Just look at how many PPL in your forum came back with comments like " wow, so INTEL didn't whop AMD's ass, they just finally caught up with AMD! Ha, ha."

Is that all they did, just catch up?

 

[magoo]

I couldn't care less if you are impressed with the Core 2 Duo's performance. Please join the conversation and keep up. We are not discussing so much the difference in CPU's and which one is better, but among other things which method of evaluating is better.

I'm paying quite close attention, thank you. You can label this anyway you wish.....you can evaluate it anyway you want...... the canned benchmark, in my humble opinion, does nothing but demonstrate you can push a button and write down a number......in this case 9% difference.
There is not even an explaination of the result until the end of the article.
Calling people "fucking retards" is useless.
Join the FS forums if you wish to be vulgar about it.

 

[GoldenTiger]

I'm paying quite close attention, thank you. You can label this anyway you wish.....you can evaluate it anyway you want...... the canned benchmark, in my humble opinion, does nothing but demonstrate you can push a button and write down a number......in this case 9% difference.
There is not even an explaination of the result until the end of the article.
Calling people "fucking retards" is useless.
Join the FS forums if you wish to be vulgar about it.

Again, you completely missed the point of the conversation. We are *NOT* discussing how well Conroe did.

 

[SamuraiInBlack]

Okay, so the general concensus feels that the [H] should include canned benches as well as their real world ones and also include even more games to the mix, with screenshots, and try not to flame anyone, but at the same time, keep what they're doing now.

Is that about the sum of it?

 

[bubbagee]

Okay, so the general concensus feels that the [H] should include canned benches as well as their real world ones and also include even more games to the mix, with screenshots, and try not to flame anyone, but at the same time, keep what they're doing now.

Is that about the sum of it?

Agreed, do both - on the one hand this and on the other that. Today you get this but tommorow you'd get that. That way no one is mislead EITHER way.

 

[Brent_Justice]

To expand: The way we test video cards is to use them in the same way that a gamer uses them, thus providing real-world feedback on performance and image quality and how video cards compare in those regards. Moving this method of testing to CPUs seems logical for today's CPUs and games.

I think we made the right decision to do this now because it really shows how CPUs compare when using it in a real-world everyday gaming situation just like you'll use it at home. Playing through games and figuring out what settings are playable on a system is what everyone does before they play a game. We do that exact same thing which tells you which video card or which CPU is better and what their relative gaming experience is like. Our method realistically represents gamers playing games.

The scientific method no longer applies to evaluating video cards and CPUs for gaming. It may apply for other types of evaluations, but not for determining real-world gaming experiences. When you want to figure out the gaming experience a product delivers there is no other way to do that IMO, and that's exactly what we are trying to represent, the real-world gaming experience.

That's my peace.

Posting this again incase it got burried in this fast moving thread.

 

[numbers]

Guys, lets forget about the conclusions made in any of the articles, and look at how the tests were performed in regards to the different groups used for testing. This was the major shortcoming of the [H]'s article. Results for previous generation chips such as the P4's and X2's should have been included. Also, different configurations of video cards should have been tested like in other articles. And to top the cake, the beginning and end of [H]'s article attacked other testers who's methodologies were more in depth and comprehensive than theirs. Again let me stress, i'm not talking about timing demos vs. gameplay, i'm talking about the scope of the hardware configurations used.

The [H] article had valid points, like this one:

"If I had an older system and had to put my foot down and choose a system with the future in mind, I would probably lean toward the Intel Core 2 Duo E6700 platform for “future proofing” if Oblivion were any indication of future games. If you have a higher-end AMD Athlon 64 system platform right now though, there really isn’t any need to go scrambling to Intel Core 2 at this particular time for gaming. I’d wait it out and see what the future brings. "

The rest of the conclusion makes similar points, albeit the overclocking comment (that goes both ways Kyle ). However, the mudslinging and narrow scope of testing have ended up overshadowing these points that should have been the focus on the article. Aside from the points I raised above, the only problem I have with this article is the black backgrounds on the fps charts

 

[RobbieV]

C'mon guys, this little rivalry seems pretty childish. So you have different philosophies on testing methods...who cares? Firing Squad has been referred to as "Firing Squid" by HOCP for years now? No wonder they responded. Making fun on another website's name seems pretty unprofessional and will surely get a response in time. Jakub's rant did not seem to have a nasty tone to it or anything, and I actually agreed with most of what he said.

I think HOCP has been way too defensive of their testing methodology. Calling everyone liars and the repeated (OLD) "canned benchmark" phrase.

Anyone been around to other tech forums? Most people do not agree with HOCP's testing procedures in this review. We read reviews to gain new information, not to find out something we already knew. I've been coming here for over a year now, although only recently started posting, and this is the first review of ANYTHING that really disappointed me. I really do not want to see this site lose a great reputation because they are too proud and can't acknowledge that other testing procedures are valid too.

 

[Menelmarar]

Posting this again incase it got burried in this fast moving thread.

That's great, and you do a good job at it.

But what about the hardware enthusiast? What about showing the CPU's strength for future upgrades. You do one peice of the puzzle very well, but what about the rest.

FS was able to show both sides, and this is only one small slide from their article that I typed some notes on. FS and many other enthusiast sites have MANY more examples of showing both sides, Real-World gameplay and enthusiast comparisons for the CPUs.

 

[Brent_Justice]

What about showing the CPU's strength for future upgrades.

We've dealt with this on the video card side of things when we moved to this new format, and so I am not surprised this question is coming up now that we've moved this over to the CPU side.

There is no way to predict the future. We'd need future games and applications for that, and they aren't there, obviously. This is the present. All we can do is realistically test the present and tell you what you will experience now. We can then make educated guesses based on our results for how things might play out in the future based on what we are seeing right now, but obviously anything can change for the future. Unless you are Johnny Smith I don't see how you can predict how CPUs will behave in games in the future.

FS and many other enthusiast sites have MANY more examples of showing both sides, Real-World gameplay and enthusiast comparisons for the CPUs.

So did we:

http://enthusiast.hardocp.com/article.html?art=MTExMCwxLCxoZW50aHVzaWFzdA==

http://enthusiast.hardocp.com/article.html?art=MTExMSwxLCxoZW50aHVzaWFzdA==

http://enthusiast.hardocp.com/article.html?art=MTEwNiwxLCxoZW50aHVzaWFzdA==

 

[lovemyPC]

Kyle,

as many others have indicated here, you've really let me down. I think you (as the author of the original article and the editorial) behaved unprofessionally and I'm disappointed in your subsequent behavior. it was and continues to be childish, unfair, and unnecessary. Do you really need to p*ss on everyone else to make your site and methods look good? I'll probably keep coming to the site, but I'm pretty disgusted right now. You've needlessly squandered a lot of the good will you've built up here.

 

[Chris_B]

I have to object to the title "Firing Squid Wastes their Ink", seeing as the articles online no ink was harmed or wasted in the making of it.

 

[magoo]

Again, you completely missed the point of the conversation. We are *NOT* discussing how well Conroe did.

Look I didnt "miss the point". I just stated the obvious. The demo benchmark you are so happy to throw up shows a 9% advantage in 800*600 benchmarks. I dont play benchmarks......I play games and do work in the real world just like everyone else.

The differences no matter how you look at them and what "flavor of test" you like to refer to are small at best.

Ive read a bunch of these reviews, like everyone else has. [H]s article started out distinctly syaing this was going to be a different kind of review, specifically looking at game results and performed in a specific way. The other results were listed elsewhere in more traditional fashion.

If you didnt like it, tough. Nobody's reviews are wrong, they're just different. Frankly, I cant see what all the hubbub is about, unless you're a !!!!!! of Intel its just good reading. There are plenty of opinions around.

 

[mrneutron2003]

This thread, relating to the banter back and forth brings to mind a back and forth debate (and I use the term lightly, as it was rather flamey at times) Kyle and I had several years ago, regarding a review I wrote on an old Abit Siluro T400 Geforce2MX400 card. In essence the problem between me and Kyle was the following..
Here's a snippet of the article

"Here I'm going to seque into a little "what-to-do-if-you're-desperate-for-a-video-card-upgrade" rant. Opinions are strictly my own.

With a reasonably modern CPU, playing today's games, there's little reason to bother with the Geforce3 at this point in time. Sure, the Geforce3 is the fastest thing on planet earth right now for gaming. But for many, not ENOUGH faster to rate the huge sticker-shock pricetag these things are garnering. For most people, playing games at typical resolutions (at or below 1024x768 resolution) on a reasonably modern (600mhz+) CPU, the Geforce2MX series of GPU's is quite adequate.

(note: after reading some follow up comments over on HardOCP.com today, I feel an obligation to further clarify my thinking, as to the above. The Following are comments I made in reply to HardOCP. Not a flame, just a clarification.

Having already witnessed the awe inspiring power of the Geforce3 myself, I still stand by that statement. The Geforce fails to be relavent enough with current generation games, to justify it's typical $250+ premium over a Geforce2mx or the roughly $200 premium over a Geforce2.

Once games that actually SHIP leverage the power of the Geforce3, (Doom3, Aquanox, Halo (assumedly this will be an X-Box first release), ect... this will change......markedly.....but we're not quite there yet.

Sure faster framerates, and usable antialiasing are available with the Geforce3 currently, but neither honestly justifies the pricetag, at least not to me. I think we're in a rather nice situation at this point in time, in 3D. First, we're finally eclipsing framerate as the end-all factor of judgement. Secondly, unlike the not too distant past, where it was either "Voodoo" or a significant compromise, we have a nice range of affordable graphics controllers, that give more than adequate performance for most people, and at reasonable prices (Kyro2, Radeon, Geforce2mx, ect). We have NVIDIA to thank largely for pressuring the market into being in the shape it's in now (with some admitted fallout, competition-wise), by continuing to push the envelope with products such as the Geforce3. Certainly there are users, who will accept no compromises, and will be willing to plonk down $350+ for a Geforce3. I would argue that the smart move for the cost-aware user would be to get one when it becomes necessary to have one to enjoy the games you play."

I was trying with the above to get beyond the raw performance, and look at this card in the contexts of 1). the resolutions most people play at and, 2). What currently shipping widely used games would be played.

It's rather funny when taken into context. My webmaster enjoyed the hell out of this, as there were significant back-and-forths in email, as well as the posting on OCP. Of course, all my webmaster ever cared about was page impressions and advertising revenue, and the *Controversy* made this one of the best page impression days of the entire year. I left this publication about 9 months later on personal and professional grounds.

To preface things, I was a relative newbie when it came to writing reviews, working within my own personal budget (I still cringe thinking how much money I ended up putting in that site webmaster's pocket over the years I worked for him, without recieving so much as a dime...dumb me for doing it), and my testing methods were pretty average, typical, and basic for the timeperiod, 2001. At the time I was beginning to experience a level of frustration with doing Hardware evaluations. Even way back then it was getting more complex than just "framerate" when it came to doing evaluations. Nvidia had successfully created multiple market tiers for graphics cards, and things like price and a products intended market segment made doing apples to apples comparisons not only hard, but basically pointless. I still cringe however looking back towards then, and realizing how "newb" I was looking at things. My article was embarrasingly basic, so it's not like I'm sitting here 5 years later trying to defend my methods. Still, this incident between us, ended up being the beginning of awareness for me of the very issue this thread covers. I think perhaps this was also the beginning inklings of Kyle's realizations too, that our work as tech publications has to evolve along with the rest of the industry.

One thing I always tried to do, was give my own personal impressions, based on my experience with a product. This is a dangerous thing to do, simply because you're going out of the objective, and into the subjective. Still, as a writer evaluating products for a reader base, it's clearly part of the job. Anyone can run 20 pages of benchmarks, point to the longest Excel bargraph line and run little risk of controversy. But this does little to serve your readership, and ends up merely being an abstract regurgitation of P.R.

If you are in this business, and really care about what you do, you will take the risks. I see it as our job as writers to step out on that limb, and try to distill all the *numbers* into something far more meaningful.

People can argue the merits of different testing methodologies back and forth till the universe collapses. Like it or not, every single independant unbiased hardware review, has a level of bias, because a human being has to evaluate things and make a recommendation. People who read online tech publications are usually savvy enough to understand the underlying data, but I feel truly appreciate the broader perspective a publication can give. We're looked upon to recommend to people what to do with their hard earned dollars, because we get to play with things before they do, so I feel a personal obligation to step outside the box and actually THINK.

I think HardOCP's testing methodologies do their best to give credence to their conclusions, and I applaud Kyle for being one of the few in this industry who are willing to think outside the box, take chances, and stand firm on their reasonings and methodologies.

 

[bassman]

Calling all other benchmarks "canned" is about as worthwile as calling his comparison "real-world".

Bingo! I respect [H] for the work Kyle, Steve, Brent, and everyone else does and the information I get from them, but calling their benchmarks "real-world actual gameplay" means nothing. It's mumbo-jumbo. It's still a benchmark.

From dictionary.com:

bench·mark
n.

1. A standard by which something can be measured or judged.

That said, I like [H]'s approach to gaming benchmarks. They help me decide how a particular system will perform in a particular game.

But I do more than game - I code, run scientific simulations, render CG, edit and encode video, etc. I don't feel I can get the whole story at [H] anymore. For example, look at the recent [H] Core 2 Duo articles. The gaming article was 11 pages long, while the media article was only 3.

I'm disappointed because I have to go elsewhere to get the whole story. I would have really liked to see CPU benchmarks in the Core 2 Duo articles, like they did in the AM2 article.

But back to the topic. To quote Grand Moff Tarkin: "This bickering is pointless. Lord [H] will provide us with the location of the best hardware by the time bassman's credit card is operational."

 

[Menelmarar]

We've dealt with this on the video card side of things when we moved to this new format, and so I am not surprised this question is coming up now that we've moved this over to the CPU side.

There is no way to predict the future. We'd need future games and applications for that, and they aren't there, obviously. This is the present. All we can do is realistically test the present and tell you what you will experience now. We can then make educated guesses based on our results for how things might play out in the future based on what we are seeing right now, but obviously anything can change for the future. Unless you are Johnny Smith I don't see how you can predict how CPUs will behave in games in the future.

I have to disagree. Assumptions have to be made obviously, and the big assumption here is that the next generation video card, DX10 type stuff on the horizon or the next nVidia/ATI refresh, will be faster than what is currently on the market. And I think we could all agree that this is a valid assumption.

With that in mind, if you can show that one CPU is better than another in gaming applications then you can come to the conclusion that once the GPU limitation is lifted that CPU platform will give you better performance which can potentially allow you to turn on better visuals and have a better gaming experience.

edit: Corrected the word "not" where it shouldn't have been, completely confusing things .

So did we:

http://enthusiast.hardocp.com/article.html?art=MTExMCwxLCxoZW50aHVzaWFzdA==

http://enthusiast.hardocp.com/article.html?art=MTExMSwxLCxoZW50aHVzaWFzdA==

http://enthusiast.hardocp.com/article.html?art=MTEwNiwxLCxoZW50aHVzaWFzdA==

While those are all very valid and good reviews Brent, and while they show strength in desktop applications and encoding they don't show gaming. How many times in the past have we see where AMD has excelled in gaming, but Intel excelled on the desktop...

 

[Brent_Justice]

but calling their benchmarks "real-world actual gameplay" means nothing. It's mumbo-jumbo. It's still a benchmark.

Actually, the proper nomenclature for what we do now is an "evaluation", not a benchmark, not a review. We don't 'review' video cards anymore for example, we 'evaluate' them, meaning we form an opinion about them, we judge, we appraise.

 

[jimmyb]

There is no way to predict the future. We'd need future games and applications for that, and they aren't there, obviously.

There are plenty of ways to predict the future. There is no way of knowing it.

I suggest you read up on data extrapolation. There is certainly uncertainty in it, but the last 500 years of technological and scientific development have proven it is often a decent indicator.

EDIT: I should add that I am working at a severe weather prediction and forecasting center this summer. We don't know what the weather will be like tomorrow, but we can make decent guesses. And weather is a lot more random compared to IT.

 

[Brent_Justice]

Isn't that what I said? We can make educated guesses based on what we know now. That's all though. Things change, and with gaming they can change day to day.

You cannot look at the gaming experience in such a scientific way anymore. This isn't the weather. You can't do it scientifically like that anymore. It just doesn't work for today's games and hardware. Data extrapolation? Come on, how is that going to tell you how each video card and CPU feels in a game? How is that going to tell you which one provides the best gaming experience? How is that going to tell what in-game settings you can enable in the game or may need to trade off for performance or IQ to get a playable game?

I'm sorry, I feel very strongly about what we are doing because it is simply the right way to evaluate the gaming experience. We play the games just like you play the games at home, our evaluation method matches that to the T.

If you know of a better way, we are all ears.

 

[foofighter06]

If you people are trying to say that a single threaded game at 800x600 today is gonna help point to which processor is better in the future, then I think your gonna be disappointed.

Directx 10 games and games that make use of 4 or more cores could react completely differently.

Again, I have zero interest in CPU test's that use app's at settings that I wont use. Period.

800x600 is pointless.

 

[Opie]

I prefer the squiggly lines and real world approach as *my* evaluation of all hardware reviews is inevitably distilled to "what is MINIMUM framerate a new part will give me at the resolutions I run my games at." Since I moved to SLi I've found that I dont care about anything less than max eyecandy results. I don't care about momentary spikes in performace resulting in a nice, long line on a bar graph. I care a little more about average frames. What I want to know, as a competitor in online leagues is what is the bottom end? What is the maximum playable setting for whatever part? Here is where I get the answer.

I am surprised, although I probably should not be, by some people's assertion that the next gen GPUs will suddenly lift the barrier and shift the bottleneck back to the CPU. This has not been the case for years. Do you know something the rest of us do not or is your personal agenda of brand X being carried through the Arc D'Triomphe simply not being met? I prefer reviews where results are arrived at by a critical eye and make the assumption that the reader can think for themselves.

I would rather not see a swordfight between websites, but hey, everyone needs a little drama in their lives.

 

[bassman]

but calling their benchmarks "real-world actual gameplay" means nothing. It's mumbo-jumbo. It's still a benchmark.

Actually, the proper nomenclature for what we do now is an "evaluation", not a benchmark, not a review. We don't 'review' video cards anymore for example, we 'evaluate' them, meaning we form an opinion about them, we judge, we appraise.

Are you being serious? Come on, Brent. Evaluate, review, judge, appraise, etc, etc, etc are all synonyms. And you need a standard of measurement (a benchmark) to do it. What's wrong with using honest, clear, direct language? I understand the need to differentiate and compete with other sites, but I think using this gobbly-gook just moves you away from your readers.

This reminds me of a George Carlin bit: "It's in the Bible: 'Jesus healed the cripples'. He didn't engage in rehabilitative strategies for the physically disadvantaged." You review computer hardware using benchmarks. (As does AT, FS, Tom's, etc.) You don't conduct a comprehensive evaluation of computing resources by utilizing real-world actual gameplay.

 

[CopyCat]

This thread is huge.

In any case, I never even heard of Firing Squid before this thread. I only ever check out the [H] and Andantech. Being here for a long time and working with computers longer so I guess I know something.

I glanced at the Firing Squid review. It not only seemed useless but wasn't well put together and didn't grab my attention. I don't care about a bunch of graphs and numbers.
[H] really hit the spot for me with their reviews. They are in depth and extremely creative. I feel like I am talking to a true gamer and seeing what they thought after running their new shit. I stopped paying attention to 3Dmark scores a long time ago.

As far as the fight going on...its not surprising. This happens online all the time. Kinda like Blog wars. I think its fun to read...and the more reason to visit the site=)

Cheers

 

[foofighter06]

Are you being serious? Come on, Brent. Evaluate, review, judge, appraise, etc, etc, etc are all synonyms. And you need a standard of measurement (a benchmark) to do it. What's wrong with using honest, clear, direct language? I understand the need to differentiate and compete with other sites, but I think using this gobbly-gook just moves you away from your readers.

This reminds me of a George Carlin bit: "It's in the Bible: 'Jesus healed the cripples'. He didn't engage in rehabilitative strategies for the physically disadvantaged." You review computer hardware using benchmarks. (As does AT, FS, Tom's, etc.) You don't conduct a comprehensive evaluation of computing resources by utilizing real-world actual gameplay.

Seems to me it would take longer to sit down and play thru a game to find which settings are best for each piece of hardware being reviewed.

Than it would take to run a timedemo in quake.

 

[evilmedic]

This "canned benchmark" vs "real world benchmark" stuff doesn't make sense at all. The benchmarks used by FS and [H] both have the following traits.

That "data" set is the same.
The "instructions" set is the same.

This "data" and "instruction set" is fed in the CPU and the results are put out into the I/O device. The big only different is that the "I/O output requirements" are very different.

800 x 600 = 480,000pixels
1024 x 768 = 768,000pixels
1280 x 1024 = 1,310,720pixels
1680 x 1050 = 1,764,000pixels
1600 x 1200 =1,920,000pixels

with the 800 x 600 tests used at FS. The CPU basically contiuously told the Graphics card to render that 480,000 pixel frames as fast as it could. Modern graphics cards are damn fast as fast CPU could feed it data to render.

with [H] the 1600 x 1200 test with FSAA + AF the amont of work the graphics cards has to do is significantly higher. So high that it chokes up and becomes the bottle neck.

Both the methods used by [H] and FS are canned. The data is the same, the instructions are the same. they are both fed into the CPU and the output is both shown on the screen.
Because of the differnt choices in output settings. One FS decided to load the CPU, [H] decied to load the I/O device.

[H]'s article, titled " Intel Core 2 Gaming Performance" was heavily focused on I/O bound scenarios.

[FS] decided to take a mix of both "Processing Power" and "I/O bound" scenarios.

There is nothing wrong with either one of these approaches. Which one is more relevent to you? Well, it depends on you want, which scenario reflects your situation.

 

[HOCP4ME]

<sigh>

How many people would like [H] to do synthetic benchmarks?

<loud cheer>

How many people would like [H] to do real-world benchmarks?

<loud cheer>

Now, how many people think that synthetic benchmarks are useless?

<loud cheer>

How many people think that real-world benchmarks are useless?

<loud cheer>

How many people think that both are useless?

<silence>



Draw your own conclusion...

 

[jimmyb]

Isn't that what I said?
[...]
Data extrapolation? Come on, how is that going to tell you how each video card and CPU feels in a game?
[...]
If you know of a better way, we are all ears.

I must have misinterpreted you. I thought you were suggesting the it was an entirely futile effort to make any prediction regarding hardware performance with future software.

We extrapolate the "feeling" much in the same way as when we look at the reviews you post, and say, "We'll I don't really play those games, but it performed really well, so I bet it will work well in the games that I do play". I'm guessing that you don't assume everyone is playing strictly the games you evaluate on.

Purchasing decisions are made based on an expectation of performance now, and in the future. Some forecasting needs to be done.

Also, standard deviation would be a great figure to include with your tests (and it doesn't really affect the underlying methodology, so I see no reason why not to implement it, even if we can't agree otherwise).

Thanks.

 

[Brent_Justice]

Seems to me it would take longer to sit down and play thru a game to find which settings are best for each piece of hardware being reviewed.

Than it would take to run a timedemo in quake.

Exactly, our evaluations are much more involved than anyone else does it, Bit-Tech excluded, they've done some damned good evals there and I know they put a lot of time into it also. This is a good read - http://www.bit-tech.net/columns/2005/10/21/open_letter_graphics/1.html

Hey, I use to do timedemos too, we use to be able to produce a review in 4-5 days! Sometimes less!

Now that we have changed to this new evaluation process the time spent is MUCH greater. When we started it the time for completion went up to 1 full week working FULL TIME, yes, 7 days straight. Now though it takes me 2 full weeks since we now added Widescreen testing into the mix and more games. So check this out, 8 games, 4:3, 16:9 testing takes 2 weeks full time work usually. Sometimes even 3 weeks.

There is just no way around it, it is much more involved now and takes more time to do it. That's why many sites don't do it, it takes time, money and resources.

Doing the same things gamers do is the only way to evaluate that gaming experience that I know of.

 

[Brent_Justice]

I must have misinterpreted you. I thought you were suggesting the it was an entirely futile effort to make any prediction regarding hardware performance with future software.

We extrapolate the "feeling" much in the same way as when we look at the reviews you post, and say, "We'll I don't really play those games, but it performed really well, so I bet it will work well in the games that I do play". I'm guessing that you don't assume everyone is playing strictly the games you evaluate on.

Purchasing decisions are made based on an expectation of performance now, and in the future. Some forecasting needs to be done.

Also, standard deviation would be a great figure to include with your tests (and it doesn't really affect the underlying methodology, so I see no reason why not to implement it, even if we can't agree otherwise).

Thanks.

Thanks for the feedback, it is nice to hear a professional and informed analysis.

 

[bassman]

Seems to me it would take longer to sit down and play thru a game to find which settings are best for each piece of hardware being reviewed.

Than it would take to run a timedemo in quake.

That's a good point, and it's why I like [H] gaming benchmarks over other sites.

I guess I haven't been clear on my point. I feel like the [H] - FS debacle is all about which foofy words are being using to describe their benchmarks. [H] started doing "real-world gameplay evaluations". Since that time the first paragraph of [H] articles have become about how their approach is better than everyone else's. (Fine, I agree, but I don't need to be told that in every article I read here.) FS latched on to those statements to start an argument and generate traffic. [H] responded, and so on, and so on.

Please don't take this negatively, Brent. I like your reviews. I just mentally have to filter them. And I'd like to see the same effort put into other components (CPU, storage, etc) as is done in video cards.

 

[jacuzz1]

You can spin it anyway you wish but the fact of the matter is that Firing Squad has been stagnating since the buy out. I lost interest in that site along time ago. I never laughed so [H]ard as when Kyle told it like it is regarding Fireing Squad's so called journalism.


The way I see it Conroe is too "big" to address with one article the [H]'s approach using a series of articles addressing real world performance in a number of catagories seems to me to be the most comprehensive, interesting and as a result, the most "journalsistic" when comparing to Firing Squad.

I was able to easily draw my own conclusion regarding what I will need now and in the future as a result and dont need to read page after page of benchmarks.

 

[Puterguru]

From the front page - "We have jokingly referred to the Firing Squad as the "Firing Squid" for years now"

Doing a GoogleSearch for "Firing Squid" brings up their website, first link.

 

[aldamon]

There is no way to predict the future.

This has probably been typed a dozen times by now, and for that I apologize, but SLI (and now XFire) has given us an excellent, dependable preview of next-generation, single-card power for the last two years.

 

[BarneyGumble]

I can see the base argument here. The basic function of journalism is to be as objective as possible. This is more easily accomplished through the proper use of "canned" benchmarks. This is also the more scientific of the two methods. However, "real-world" benchmarks are more useful to the end-user of a product.

 

[rodman]

I like [H] benchmarks myself. I could care less about there use of canned benchies simply becouse most other sights have these. The one thing the [H] should change is when comparing diff CPUs and GPUs using real world tests, is use the SAME graphics settings. I get confused when looking at MIN and MAX frame rates between diff hardware using diff graphics settings as sometimes they look the same on a graph but thats becouse diff graphics settings were used benching the same hardware. It's much easyer to understand a benchmark graph between diff hardware tested at the same graphics setting. I have no idea why [H] does not do this..

 

[HighwayAssassins]

Kyle, I just read your retort.

I feel mixed on the issue, as I like and will search out both forms of review on any product I buy. That said, I certainly agree with you about "Canned Benchmarks". I always skip passed the "synthetic" benchmarks, because they truly mean nothing. So what if that computer scored whatever PCMark score? I am not buying a computer to run PC mark all day. Let me see some gaming info!!

For this website, I think most people have high end computers, because this site caters to the enthusiast. Not everyone here has a top of the line rig, but many do. These are the kind of benchmarks we need at the H. if Firing Squad doesnt like it, they dont have to read the articles.

I dont know why people are so pissed off at you. You are known for stating how you feel and standing by it. I respect that, and find it beneficial to the site (except maybe the intelitxt ), so keep it up, its why im a [H] member, not a Firing Squad member, although I have nothing against them.

[H] just caters to my personal computing needs.

 

[bingo13]

Actually, the proper nomenclature for what we do now is an "evaluation", not a benchmark, not a review. We don't 'review' video cards anymore for example, we 'evaluate' them, meaning we form an opinion about them, we judge, we appraise.

Then why does Kyle rant and rave that everyone is a liar or cronie if they have a different opinion or conclusion about the product? Really, if the product is not reviewed then why is your opinion any better than mine?

 

[Cleeve]

How can HardOCP claim to not be creating benchmarks?

It's the same thing with diferent methodology. Firingsquad's method shows difference in platform capabilites by lowering resolution, and HardOCP shows differences in platform capabilities by lowering settings.

In both cases a variable is changed to get a desired result. The only difference is, HardOCP's has the advantage of setting itself apart from the pack, but to be honest I find it a bit less intuitive.

But for HardOCP to diss everyone else's IMHO more scientific method of comparing kit, is elitist and adversarial. What purpose did that serve?

Is it truly that difficult to see the value in lowering resolution to see where the CPU bottleneck lies? Is anyone truly convinced that benchmarking is completely useless?

Honestly guys. Do we really need head butting and chest thumping? This doesn't seem very professional to me.

 

[HighwayAssassins]

Honestly guys. Do we really need head butting and chest thumping? This doesn't seem very professional to me.

Thats what im saying. not only on the professional side, but also the forum side.

I cant believe how many words have been written about one measly revi- ah, I mean evaluation. Just give it a rest people!! dont read it if you dont like what you see!

I mean, is there anyone here who is interested in buying a Conroe and only looked at kyles writeup? I bet not. Its just another article to be taken with all the others to form one complete picture.