Bit-Tech Drops Real World Testing

[AxeBlade]

I love the real world testing. I have never read BT, but it is to bad that another site has left this form of testing eventhough I know it's probably time consuming.
[H] rocks.
Hey Kyle. If Brasso works for balls, do you have anything for my old mans bald head.

 

[Catweazle]

So three weeks is too long to spend writing a review on an item that gets updated every 6 monthes?!? It sad really given that form me as a consumer there two key advantages of a website over a magazine: speed and more in depth coverage. The magazines just can't devote the space or time to do tests, reviews or for the most part articles in any kind of meaningful way. It is the time put into and the greater detail that make tech websites worth reading.

We don't need to dumb it down. We have CNET for that and the last I checked Joe Six Pack doesn't get beyond that or PC World.

It's obvious to me, as a bloke who writes IT stuff for payment, that your analysis and understanding is a bit out of whack.

Print media, whilst being definitely being subject to space restrictions, is not subject to time restrictions in the same way as web media. Ot's an established 'given' that articles in print media will appear later than articles appearing in web media, and the magazines are competing, within that space restriction, to produce the highest quality content. (Payment for print media articles is generally higher than payment for web media articles, by the way.) Writers have longer lead times, lead writers generally have better test labs at their disposal, and whilst articles will inevitably be shorter in length on average they'll often have had more extensive work done to produce them.

In contrast, web articles are predominately competing to be the 'first' to appear. Time constraints rather than space (or in some cases quality) constraints become a governing factor. Have your article be one of the first of its type to appear and you'll generally progress up the search rankings, attract more page views, enjoy more advertising revenue etc etc. Take longer to get that article published and, to a large extent, it won't matter much that it's 'better' than a rival article. The fickle search engine audience will respond with "Old news!"

Print magazines and websites are both still "publishers", but they're very different environments, and subject to differing constraints.


And, by the way, I have a question. If the techie websites are paying so much attention to detail and producing work which is so much more meaningful, how come we still have so many regular contributors to techie forums who make such broad, sweeping generalisations which suggest that 'bigger numbers' must necessarily mean 'better in every way'?

 

[Catweazle]

Resurrecting this, and for good reason. I'm confronted with a question and I'm coming up short on answers.

Two things happen in the marketplace when a new graphics card generation is released. People get busy buying the new cards. People get busy selling the old ones. And with the change to 'real world benchmarking' the answers just aren;t there for the people next in line down the pecking order - the people buying up those secondhand cards.

I'm sitting here confronted with just such a dilemma. A system with a pair of SLI'd 6600 cards, and a choice to make. A couple of 7800GT cards on offer at a very affordable price. Decisions to make. Sure, a current generation card could be had for a bit more monetary outlay, but DX10 games in all their glory aren't really going to be a reality on this rig until today's generation of cards are somebody's cast-offs, and the extra money puts it out of reach anyway.


So, to the point of all this:

Those 'real-world' benchmarks don't help in the slightest! Guess what? The 'real world' games those 7800s were benchmarked on aren't the same as the 'real world' games the 8800s are being tested on. The 'real world' games the 6600s were tested on are different again. there's no means, amongst the mountains of very decent technical reviews available, to get a really good picture of cross-generational comparison. Guess what's left to fall back on as far as cross-generational comparisons go? Yep! Good old 3DMark! It's far from a good guide to performance, but it's no wonder some people cling to it regardless.

Even if spending the extra dollars to get a 'bit less than the best' current day card was a consideration the benchtests being run don't really help the decision in this scenario. Who cares how well it's gonna handle the demands software throws at the hardware six months down the track when the point of the exercise is to ensure that it creams the demands software threw at the hardware six to twelve months ago? That comparison simply isn't there in the 'real world' testing, I'm afraid.


The people tacking 3DMark results onto the real world results aren't really meeting that need either. They just somehow seem to be the only ones making any semblance of effort to do so.

Pretty decent sized market out there for when todays goodness becomes tomorrow's cast-offs. Those people are 'enthusiasts' too. They just sit a tad down the pecking order, is all!

 

[Luthorcrow]

It's obvious to me, as a bloke who writes IT stuff for payment, that your analysis and understanding is a bit out of whack.

You have a point.

...print media...Writers have longer lead times, lead writers generally have better test labs at their disposal, and whilst articles will inevitably be shorter in length on average they'll often have had more extensive work done to produce them.

Where are the results? I am not claiming to know how much better the labs are at "PC World" or "Maximum PC" as compared to say [H]ardOCP, Anadtech, etc but I do know that there is a major difference in the quality of the reviews. Is it possible the writer for the print magazine did "more extensive" work (research)? Maybe but if none of that data is published in the article, methods detailed, etc then what difference does it make? If we don't know exactly what they did to test, their results, etc then what do we really have? I am not saying that every tech website is better than ever print magazine. But I would say the best tech sites are by far superior to the best of the PC Magazines. It all goes back to the point we agree on, space. How much information can you really put in a 500-2000 word article vs. what a good website can produce?

...And, by the way, I have a question. If the techie websites are paying so much attention to detail and producing work which is so much more meaningful, how come we still have so many regular contributors to techie forums who make such broad, sweeping generalisations which suggest that 'bigger numbers' must necessarily mean 'better in every way'?

If I made that statement maybe I could. Its not "bigger numbers" its more detail. It is a simple as this, when presented with a claim, how can you fact check an expert that doesn't share how they arrived at a conclusion? Whereas if the same expert gave detailed references and facts to support their claims, assuming they check out, wouldn't you be more inclined to accept the conclusions of the second expert over the first? To often when reading a PC Magazine, you are asked not to look "behind the curtain."

 

[Cyrilix]

I prefer to read FPS benchmarks, and make the analysis myself through the benchmark numbers and FRAPS graphs.

[H] gives you a different story, which I definitely do not prefer, however, I read through all the reviews anyways and [H] is one among many. As I see it, what you guys call "real-world benchmarking" will never be a standard, because it is not a standard. It relies on interpretation on how to run the benchmark -- how accurate is your playthrough going to be? How careful have you been throughout the entire benchmark to do exactly the same thing when you tested with the other card and how are you going to guarantee that you're paying the same amount of attention this time around vs. last time (you might be more tired the second time around, and notice less details, be less impressed, etc.). A significant amount of human bias comes into this kind of testing and this bias is entirely avoidable, which is why FPS benchmarks are the standard and will always remain the standard. They rely on only a two sources of error, the software, and the hardware. When you add a 3rd layer into this pool of errors, things become wacky and wild.