Bit-Tech Drops Real World Testing

That was awesome. but it doesn't explain the link you posted 0_0


Hehe, sorry, I thought I was making a joke. He mentioned "polishing my ego," and Brasso is brass polish and I thought that and the ego thing went right to brass balls...maybe I just have balls on my mind. :eek:
 
and what of jenkins mctwentypacs like myself?

it doesn't take sobriety to comprehend a [H] review. just basic grade school english.


oh, right....



there is so much more information to be gathered from a concisely written paragraph accompanied with a graph than from a colorful graph which has a sentence below describing only the outcome of the graph.

i want to know why. [H] lets jerry streetweed know WHY... assuming they didn't cheat on their grade 6 book reports.

If you think you are a "joe sixpack" and are reading up on how exactly to what detail alpha textures are aliased, I got something to tell you, you are not "joe sixpack." I am not saying you have to be a rocket scientist to comprehend the material, I am saying you have to have an innate passion for the technology behind it. And that sir, Joe Sixpack does not have.

And while you are at it, can you hook me up with Mr. Streetweed? It has been a while since I talked to him. :p
 
Kyle, been hanging around here for years before my join date. Only two things that'll make me give up PC's and the internet, whichever comes first: the day I can't buy a CRT or the day you sell the [H]. Until then, keep doin' it your way.

(Hopefully, you'll outlast the availability of CRT's.:p )
 
Real world testing falls far short of what can be provided by extensive direct comparison benchmarking -- period. I can see where real world testing could be beneficial, but that's only when doing a limited number of tests (more specifically, one given test per game). Personally, I prefer Anandtech's reviews (I only use this site for news anymore -- which you now seem to be just recycling over and over... ). Nothing beats their extensive reviews; which start with a several-page well-written editorial covering the new technology, and end with a dozen or so more pages of extensive benchmarking, with multiple tests per game and at several levels of AA or AF.

Fact is, if you guys want to hack together a quickie 2 page article to be the first ones to the press, real-world benchmarking is about the only way you could come out with a decent review. But if you want to take the time to create a truly insightful and informative article, your going to have to do a little more than one canned benchmark each from a handfull of games.
 
IMO, the 'real world' testing let's the gamer know what settings he can expect to be able to run at with the specified game.

the benchmarking that Kyle dislikes tells the game developer how to structure his code to make best use of the card.

In short: Kyle, get off your high horse. There is no wrong way to benchmark (there is no right way either).
 
IMO, the 'real world' testing let's the gamer know what settings he can expect to be able to run at with the specified game.

the benchmarking that Kyle dislikes tells the game developer how to structure his code to make best use of the card. It also allows comparison between specific subsystems of the card.

In short: Kyle, get off your high horse. There is no wrong way to benchmark (there is no right way either).
 
Yes, there is a wrong way to do everything. Some more than others. And you did not have to say it twice. :p I am just glad we can offer folks an alternative to the cookie cutter. I am sorry you do not realize the value in what we do around here, but [H] is certainly not for everyone. We will continue to link you to other hardware sites around the Web that do it differently.

Thanks for sharing.
 
Whenever [H]ard|OCP posts a video card review, I'll read through every single page, as I enjoy reading about what settings I can realistically expect to run my games at with a new graphics card. This is especially true now that games have a ton of graphics options to improve quality (Oblivion/STALKER/FSX.) I rely on [H]ard|OCP's testing to at least give me a baseline, since I'd prefer to install a game and get right down to playing it as quick as possible. In other words, thanks guys for having a heck of a lot more patience than me when it comes to this. :)

Another thing: Quoting a single number to indicate performance is absolutely meaningless. Oh sure, card X gets 75fps at a resolution of 1920x1200 -- but how often does it drop below 30fps? Yes I understand that if card Y has an average of 100fps, then there's a better chance of staying above 30fps, but it's no guarantee. Perhaps Card Y has a problem rendering smoke, so frames with heavy smoke would run at 10-15fps, versus Card X only dropping to 25-30 during such instances -- you just cannot tell that with one single number. Even showing a single "min/max/average" number doesn't alleviate this - you need a graph showing the framerate over a period of time like [H] does to get the true picture.

Bottom line? Keep up the great work guys. :)
 
Yep, the site targets the technically minded. Does a fair job of it too, in my opinion. But no need to 'diss' on Joe. Joe's cool. He shouldn't be dismissed or disparaged just because people don't know how to speak his language!

Let my try to say this again and get it right. Our Joe Sixpacks might be incredibly intelligent individuals, the simply do not have the innate desire or passion towards understanding the tech we discuss. I am not dissing them, I am just saying they really give a shit what we have to say about alpha texture blending. They just want to game. :)
 
From the Article:
Tim would be writing the article for three weeks if that was the case.

Yes, that's absolutely correct.

I believe my average came out to around 4 weeks per computer when I was evaluating for HConsumer, although admittedly, not all of that was game testing.
 
Real world testing falls far short of what can be provided by extensive direct comparison benchmarking -- period. I can see where real world testing could be beneficial, but that's only when doing a limited number of tests (more specifically, one given test per game). Personally, I prefer Anandtech's reviews (I only use this site for news anymore -- which you now seem to be just recycling over and over... ). Nothing beats their extensive reviews; which start with a several-page well-written editorial covering the new technology, and end with a dozen or so more pages of extensive benchmarking, with multiple tests per game and at several levels of AA or AF.

Fact is, if you guys want to hack together a quickie 2 page article to be the first ones to the press, real-world benchmarking is about the only way you could come out with a decent review. But if you want to take the time to create a truly insightful and informative article, your going to have to do a little more than one canned benchmark each from a handfull of games.


Anandtech ahs the worst video card reivews. They give you zero damn details abotu in game settings and they do not usually give you minimum frames, which are the most important in my book.
 
IMO, the 'real world' testing let's the gamer know what settings he can expect to be able to run at with the specified game.

the benchmarking that Kyle dislikes tells the game developer how to structure his code to make best use of the card. It also allows comparison between specific subsystems of the card.

In short: Kyle, get off your high horse. There is no wrong way to benchmark (there is no right way either).

That's some mamby pamby lets all be feinds crap right there. There is absouluty superiour ways for testing cards, just as there is superiour ways of testing just about anything. Giving the average FPS for a bunch of canned benches is worthelss. I don't set my card up so they runs at over 100fps when playing, I crank that shit up till my card begs for mercy. I gotta know what the minimum frames are and how long it stays down there for. Anything less is worthless pap.
 
I love the [H] reviews so keep em coming.

My only comment on BT is that they couldn't have picked a worst time to change their testing methodology; and on such a controversial piece of hardware no less.
 
Whilst I agree with that in large part, I don't necessarily agree that it's all beyond the comprehension of the average user. That's predominately in the telling of it, and techies often get so caught up in the technicalities of the technical terms that they fail to communicate to anyone other than those similarly equipped with technical expertise, and assume that the reason is because "Joe Average" is of lesser ability to comprehend. It's been my experience and feedback, from the past 3 years or so of writing specifically for a 'Joe Average' audience, that if you discard the technical jargon and instead talk in more everyday language, Joe is usually capable of absorbing far more than he's given credit for. Bombard a bloke with jargon and you'll usually be met with a blank stare. Get rid of the jargon and use the bloke's own language and the reaction can be quite different indeed.

Here is a good idea Kyle. Since the big push these days is for the sub $100 laptop to help third world country students, why not have Babelfish translate your reviews into ebonics to help educate kids in third world countries such as the "inner city", "south central" and "compton".. LMAO


Seriously though. This site has always been geared toward the hardcore computer enthusiasts. Translating this site from geek speak to Grandma Jo speak would kill the essence of what Kyle strived to achieve. Not everyone here comes in with all the knowledge and lingo, but if they really want to learn the ins & outs of the technology there are plenty of resources available online that will fill that need. That's what google and wiki are for.
 
@ SpoogeMonkey

(bugger the inability to post edit!)


My comments weren't suggesting that the site should be changed to 'Grandma speak'. they addressed two things only:

* Joe gets 'dissed' undeservedly
* Hardware sites in general (and [H]ard isn't necessarily an exception) often fall short insofar as drawing the connections between the numbers and the practicalities.
 
I like both types of reviews. I dont see why you really cant have both worlds. apples oranges, and grapes. They all taste good, why cant I eat grapes, apples, and oranges at the same time? If I had to look at every stupid oranges to apples crap I would get bored, I am glad there are sites that do it straight up. I am also glad there are websites that compare highest playable results.
 
Apples to apples is by far a better way to review a card if what you want is performance numbers and comparison.

So then how many people will be buying an x2900XT over a GTX because 3dmark says it is faster, when in real world gaming it is not?

Then when they say "well Bit-tech review showed it as being faster, but my games arent"

bit-tech going to say, sorry, not our fualt we are too lazy and want to get our review out to everyone faster to get more hits....
 
I flat out asked bindibagi on there forums, why it was dropped, and replied by saying they couldnt be bothered as it was too expensive. i think there is every right to discredit what they do now... I hate to stir up trouble, but that is just dishonest if i've ever seen it. it's a dis service to the readers over there, and it seriously irks me.

There previous reviews were my only source of backup towards [H]'s results to reassure me as to what i felt, and they dropped them because they couldn't be bothered. its disturbing...


Thank you Kyle, for not being a sell out. *slautes*
Hi 0mega,

Just to correct what was actually said (you obviously chose to read what was said differently and then sensationalised it). We said that it is physically impossible for me to shut myself away for days on end to test graphics cards for a comparison because I have taken over the role of Editor at bit-tech.

My phone rings, hourly, and as I am sure Brent and Mark can testify, real world comparisons require a hell of a lot of concentration. You are the "demo" and it's being run by you, not by the computer. You don't see Kyle writing these reviews, do you? That's because I'm sure his phone like mine, rings a lot.

So, it's not that I can't be bothered, it's that I can't afford the concentration - I still have the same time available, it's just hands off time rather than hands on. I could continue doing real world testing with a considerable sacrifice in the quality (and depth) of the reviews or I could opt for a more hands off approach. I chose the latter because I've never agreed with the former. HardOCP is also a bigger organisation than bit-tech too - Kyle can afford to throw money at two dedicated graphics card reviewers, I do all of bit-tech's graphics reviews myself.

In the past, a launch review would take me two to three weeks to complete and this particular one actually took me longer to complete. The method I'm using now is more hands off, but I still sit there while every benchmark is being run, often to the very early hours of the morning. It's not that *I* can't be bothered, it's that it is physically impossible for me to do my job and write reviews in the same way that I did in the past. Believe me, I am still a firm believer in real world testing and if I could shut myself away for days on end, I would.

There is also the fact that in a recent reader survey we conducted, the majority of the readers that responded said that they preferred apples to apples benchmarks. We actually implemented that back in February, so R600 isn't the first review we've done with "apples to apples".

On to the review itself... While I didn't quite come to the exactly same conclusion as HardOCP (in that R600 is a flop - which I personally think is a bit strong in its early days), I did state the following:

Unfortunately for AMD, R600 just isn’t that special because not only is Nvidia’s performance crown still intact, the card AMD has chosen to attack – the GeForce 8800 GTS 640MB – has come away with all but a few chinks in its armour.

That’s not to say that R600 is a particularly bad card, it’s just not going to set the world on fire (well, it depends which way you look at it – Ed.) when it comes to performance. There are undoubtedly scenarios where it does perform fantastically and much to our surprise one of those scenarios is OpenGL – a territory that was once an Nvidia fortress. Not anymore, it would seem.

I only used one of the games that Brent used in his review: Oblivion, and I had this to say about R600's Oblivion performance:

Performance in Oblivion started off really well without anti-aliasing enabled, but once anti-aliasing was appled, Nvidia's GeForce 8800 GTS 640MB blitzed past the HD 2900 XT. Even worse is the fact that the Radeon X1950 XTX manages to outperform the Radeon HD 2900 XT at 1600x1200 4xAA 16xAF. Because ATI was touting HDR-plus-AA as the only way to play Oblivion, it appears to have come back and bitten its backside.


What I will say to end is that I've come to virtually the same conclusion, stating that it isn't a recommended buy (we gave it 7/10 overall - an 'average' score) and that those with GTX or GTS cards should definitely not be disappointed.
 
I actually prefer apples to apples over 'real world testing', but do like to take both in consideration.

The only benchmarks that matter to me anymore is how a card performs at 1920x1200, and for now i am holding out until the 8900GTX at a minimum before I make my DX10 upgrade purchase.
 
Hi 0mega,

Just to correct what was actually said (you obviously chose to read what was said differently and then sensationalised it). We said that it is physically impossible for me to shut myself away for days on end to test graphics cards for a comparison because I have taken over the role of Editor at bit-tech.

My phone rings, hourly, and as I am sure Brent and Mark can testify, real world comparisons require a hell of a lot of concentration. You are the "demo" and it's being run by you, not by the computer. You don't see Kyle writing these reviews, do you? That's because I'm sure his phone like mine, rings a lot.

So, it's not that I can't be bothered, it's that I can't afford the concentration - I still have the same time available, it's just hands off time rather than hands on. I could continue doing real world testing with a considerable sacrifice in the quality (and depth) of the reviews or I could opt for a more hands off approach. I chose the latter because I've never agreed with the former. HardOCP is also a bigger organisation than bit-tech too - Kyle can afford to throw money at two dedicated graphics card reviewers, I do all of bit-tech's graphics reviews myself.

In the past, a launch review would take me two to three weeks to complete and this particular one actually took me longer to complete. The method I'm using now is more hands off, but I still sit there while every benchmark is being run, often to the very early hours of the morning. It's not that *I* can't be bothered, it's that it is physically impossible for me to do my job and write reviews in the same way that I did in the past. Believe me, I am still a firm believer in real world testing and if I could shut myself away for days on end, I would.

There is also the fact that in a recent reader survey we conducted, the majority of the readers that responded said that they preferred apples to apples benchmarks. We actually implemented that back in February, so R600 isn't the first review we've done with "apples to apples".

On to the review itself... While I didn't quite come to the exactly same conclusion as HardOCP (in that R600 is a flop - which I personally think is a bit strong in its early days), I did state the following:



I only used one of the games that Brent used in his review: Oblivion, and I had this to say about R600's Oblivion performance:




What I will say to end is that I've come to virtually the same conclusion, stating that it isn't a recommended buy (we gave it 7/10 overall - an 'average' score) and that those with GTX or GTS cards should definitely not be disappointed.

I do indeed need to apologize for the couldnt be bothered bit, it was a little harsh, but i was kinda ticked as your previous reviews were awesome, and i was getting tired, Sorry about that :(
 
Hi 0mega,

Just to correct what was actually said (you obviously chose to read what was said differently and then sensationalised it). We said that it is physically impossible for me to shut myself away for days on end to test graphics cards for a comparison because I have taken over the role of Editor at bit-tech.

My phone rings, hourly, and as I am sure Brent and Mark can testify, real world comparisons require a hell of a lot of concentration. You are the "demo" and it's being run by you, not by the computer. You don't see Kyle writing these reviews, do you? That's because I'm sure his phone like mine, rings a lot.

So, it's not that I can't be bothered, it's that I can't afford the concentration - I still have the same time available, it's just hands off time rather than hands on. I could continue doing real world testing with a considerable sacrifice in the quality (and depth) of the reviews or I could opt for a more hands off approach. I chose the latter because I've never agreed with the former. HardOCP is also a bigger organisation than bit-tech too - Kyle can afford to throw money at two dedicated graphics card reviewers, I do all of bit-tech's graphics reviews myself.

In the past, a launch review would take me two to three weeks to complete and this particular one actually took me longer to complete. The method I'm using now is more hands off, but I still sit there while every benchmark is being run, often to the very early hours of the morning. It's not that *I* can't be bothered, it's that it is physically impossible for me to do my job and write reviews in the same way that I did in the past. Believe me, I am still a firm believer in real world testing and if I could shut myself away for days on end, I would.

There is also the fact that in a recent reader survey we conducted, the majority of the readers that responded said that they preferred apples to apples benchmarks. We actually implemented that back in February, so R600 isn't the first review we've done with "apples to apples".

On to the review itself... While I didn't quite come to the exactly same conclusion as HardOCP (in that R600 is a flop - which I personally think is a bit strong in its early days), I did state the following:



I only used one of the games that Brent used in his review: Oblivion, and I had this to say about R600's Oblivion performance:




What I will say to end is that I've come to virtually the same conclusion, stating that it isn't a recommended buy (we gave it 7/10 overall - an 'average' score) and that those with GTX or GTS cards should definitely not be disappointed.
Sorry about the double post :p

But i do have to add more, i did not know you had taken over as lead editor for bit-tech, had i known that now, it probably would have made more sense to me, but it still is a disservice to readers IMO... they just aren't as useful as quality comparisons... and i know self run benchmarks are a bitch. getting a route setup and keeping it from changing is a bastard. i think macros that worked would make everything easier lmfao. distractions of the slightest grade screw it up big time :p
 
I've always valued real-world testing, which I even feed to my customers.
I don't care what the video card can do but what I can do with the video card.
 
I have to sympathise with Bit-Tech some-what, but I don't totally agree with their decision to drop real-world testing.

The reason I sympathise is because I REALLY wanted a real-world comparision between the 7600GT (that I've got) and the 8600GTS, but I understand there's not enough time to do every single comparison possible, and the site has to cater for the needs of the many first.

However, I still feel they should keep them, as it's a better gauge of performance, as HardOCP has proven over the years.
 
I actually prefer apples to apples over 'real world testing', but do like to take both in consideration.

The only benchmarks that matter to me anymore is how a card performs at 1920x1200, and for now i am holding out until the 8900GTX at a minimum before I make my DX10 upgrade purchase.

I agree, give me 1900x1200 at max settings apples to apples. :)
 
Here's a question that I haven't seen pop up in this thread yet, but immediately comes to mind when I read a story like this.

[H] is, obviously, pushing the boundaries when it comes to their philosophies on what a review site should be, do, contribute. Now, it seems, other sites just can't hang with [H]; be it monetary, equipment or staffing issues.

So this is to the [H] Staff --Kyle, Brent, Steve, Dan, Mark, Brian, Justin, Rich, and anyone I left out; What can we, as your readers, do to support your endeavors?

I know there's no way to guarantee success or long prosperity, but I'm sure that if all of us who believe in your approach knew someway(s) we could give back, we would.

I currently own 3 Ratpadz (not sure if that directly or indirectly helps, but I love my ratpadz!) and have bought a few others for family/friends getting new hardware. I also try to click on any ads that may/may not be of interest to me. Would a subscription directly help? Is there any more we can do; i.e T-Shirts, Stickers, etc. Or is it really all up to the sponsors at the end of the day?
 
I love this site but would never pay for subscription. I think all your other ways of supporting the HardOCP crew is viable.

By the way... way too go [H]!
 
I currently own 3 Ratpadz (not sure if that directly or indirectly helps, but I love my ratpadz!) and have bought a few others for family/friends getting new hardware. I also try to click on any ads that may/may not be of interest to me. Would a subscription directly help? Is there any more we can do; i.e T-Shirts, Stickers, etc. Or is it really all up to the sponsors at the end of the day?

The best thing a reader can do for HardOCP outside of actually reading our pages and allowing us banner ad serves is to click the ads. If you are blocking our ads, you are circumventing the only way we get compensated for all the work we publish so that you can read it for free. Using our ads and clicking on them ads is a huge benefit to us because it is the one proof point to our advertisers that their ads are gaining the attention of our readers. And especially if you need to buy something say from Zip Zoom Fly or Newegg, please click through our banners to get there, as they do very much track those sales.
 
Would a subscription directly help? Is there any more we can do; i.e T-Shirts, Stickers, etc. Or is it really all up to the sponsors at the end of the day?

Also as to HardForum subscriptions, all that money goes back into the forums. You guys just got a new $6000 uber-database server to hopefully speed up searches and keep us running a bit more efficient in here.
 
I run Firefox 2.0 as my main browser of choice, so it's the one that accesses [H] the most. Having said that, I do rely heavily on a great add-on called No-script for some security protection (apologies if this breaks any forum rules, delete me if necessary), BUT, with sites I come to trust and rely on I do allow any script (read: ad) to run globally on those sites -- as a budding web developer I understand and respect the need for click-through advertising, just wish it was more standardized.

The moment I install the browser and the moment I navigate to [H]OCP I let all your you ad-click banners though, as well as click the actual banners when ever I remember to purchase something from one of your supporting vendors; now I probably don't click the banners _everytime_ I need to purchase something, but I'll make sure to from here on out.




Good enough! As soon as this post is submitted, I'll be entering my CC info. ;)


Edit: Rather I'll be entering my Paypal stuffs. :p Cool, now I can Edit in the Front Page News section! That was fast!

I love this site but would never pay for subscription. I think all your other ways of supporting the HardOCP crew is viable.

By the way... way too go [H]!

@ $5.00 USD for 1year and $15.00 USD for lifetime, is it really that much? I've wasted more money on less ROI that that.. Supporting the people you think are worth it, is always(sometimes) worth the value!
 
There is also the fact that in a recent reader survey we conducted, the majority of the readers that responded said that they preferred apples to apples benchmarks. We actually implemented that back in February, so R600 isn't the first review we've done with "apples to apples".

Hey as long as you state the minimum frames and give a detailed account of exactly what IG settings you're using, you'll still be head and shoulders above the rest of the pack.
 
I like the newest [H] review how it used both methods to compare. It's nice to see BOTH... otherwise if you only get apples to oranges, you may see "both get the same fps, but this one lets me enable grass".. that doesn't really tell me that much. It may be worth saving $200 if all I lose is grass.. but the apples to apples helps to show some other interesting stuff.
 
It is sad that people who can't seem to think, or posses a modicum of reading comprehension make up the majority of viewers of bit-tech's articles. For those readers, having to carefully read to see which computer part allowed for the highest level of settings for a particular game appears to be beyond them. Maybe if the reviews included a giant smiley face next to the fastest rig and a giant sad face next to the slowest those readers on the short bus might have a better chance of understanding the articles.
 
Everyone will see the light once the {H} way of testing becomes THE standard. It's rather simple really.... Games in the past are not as dynamic as they are now, period. To still rely (keyword: rely) on canned benchmarks means you are still relying on age old methods of doing stuff. That's almost as bad as following the bible.(sorry had to throw that in there).

Basically anyone who buys an HD2900XT right now is not very smart or doesnt care about his money that much. The person that see it beating the GTS's will jus tbe disappointed when he starts to game on it. The H is just trying to save the customers and Joe schmoe the restocking fee.

Oh yea, and 6 months late to the game? That's a whole product cycle. To see how bad that is, just think what would happen to NV if they never released their 7900 series. They were 6 months late and only had the 7800 to show for it?
 
So three weeks is too long to spend writing a review on an item that gets updated every 6 monthes?!? It sad really given that form me as a consumer there two key advantages of a website over a magazine: speed and more in depth coverage. The magazines just can't devote the space or time to do tests, reviews or for the most part articles in any kind of meaningful way. It is the time put into and the greater detail that make tech websites worth reading.

We don't need to dumb it down. We have CNET for that and the last I checked Joe Six Pack doesn't get beyond that or PC World.:p
 
its all about the aa people. not the regular aa, but the transperency aa that tells the tale. the apples to apples benchmarks comparing the previous generation premium cards, the 1900xtx and the factory overclocked 7900gtx showed the ati card regularly producing as much as 20 frames more than the nvidia card. this was with standard aa and aniso turned on. as soon as you switched on adaptive aa and trasperency supersampling aa the frame rates even out. the ati card did however produce a prettier picture by virtue of its hq aniso capability. in the 2900xt review vs the 8800gts, not only does the nvidia card produce more grass (anyone who has actually played oblivion knows how much of a performance hit grass takes) but it is able to utilize transperency aa over the regular aa employed by the ati card. only through rigorous, time consuming real world test can reviewers illuminate the discrepencies between the performance capabilities of both cards.
 
Haven't read the whole thread but for sure [H] has the BEST vid card reviews hands down. They do in depth analysis of games at the playable resolutions, where your frame rates will not be unplayable, as well as articles on important things like CPU scaling.

I always see CPU reviews and they look at games run at 1024x768 (previous 640x480) and I think "Who cares? Who runs at game at 1024x768?".

It's a shame that BT is moving away from real-world performance testing. I remember when [H] made all the noise about the benchmarking issue a few years ago, with 3dmurk and the other optimizations, and they didn't just bitch and moan about it but came up with a great system for testing: real world performance. Maybe it takes longer, but it also delivers the most relevant results not just for enthusiasts but every PC gamer.
 
Back
Top