Bit-Tech Drops Real World Testing

FrgMstr

FrgMstr

Just Plain Mean
Staff member
Joined
May 18, 1997
Messages
55,626
One of the leading hardware enthusiast websites in the UK is dropping their real world gameplay comparisons noting resource issues and reader disinterest. I can tell you this, Bit-Tech did a very good job of laying out why they were changing their video card testing approach back in 2005, but I guess they are having a change of heart as stated by Bit-Tech staffer Richard Swinburne.

Q: Where's the apples and oranges stuff you guys used to do? It was the best indication of how performance scales at quality and the only thing I personally feels how a GPU really performs in the real world. Benchmarks and FPS graphs just don't do it for me.
Click to expand...
 
I can't say that I ever read Bit-Tech's video card articles, but I'm sorry to see them dropping real-world gameplay analysis.
 
Hurray for real world testing,

The same people who follow fps testing are the ones that go to bestbuy for their hardware needs ;/


Jokes aside, another real guide for the testing that Hard OCp does is that it allows me to easily fine tune machine with simliar spec for best game play experience.
 
Mark_Warner said:
I can't say that I ever read Bit-Tech's video card articles, but I'm sorry to see them dropping real-world gameplay analysis.
Click to expand...


They did an excellent job at it. They were always worth the read.
 
I'm in both camps.

I NEED my apples to apples comparison so I can really quantify just how much better a card is at a given setting. That is where I make my decision on buying a card. B/c I still play most of my games at 1152x864 with as high of AA/AF I can do and still get good frame rates.

That said, the apples to oranges comparisons are growing on me. But I prefer when the in game settings are left the same (or VERY close). And only resolution and AA/AF are adjusted. If a lot of in game setting are changed its just too many variables to compare. Its good to see whats the max setting you can use for a given card/game.
 
I'll third that, I always enjoyed their evaluations. Sad to see this happen, it is really a testament to how involved the processes is, it takes a lot of resources other sites simply do not have, so they resort to taking the easier way out.
 
chrisf6969 said:
I'm in both camps.

B/c I still play most of my games at 1152x864 with as high of AA/AF I can do and still get good frame rates.
Click to expand...


One day you will start digging on high resolution textures and you will never look back. ;)
 
I am also sad to see them go back to the "darkside". Hopefully we pick up some of their readers who just can't go back to canned benchies. I for one, could never go back.

So does this make us the only site doing it the right way? :eek:
 
Going to miss their reviews very much. Times change but this change is too bad :(
 
I haven't been a PC "graphics ho" for a long time but I'm consistently amazed by the negative reaction people have to the real world comparisons. Maybe I shouldn't be, though. People do tend to like complicated issues boiled down to their simplest form, even if it has a tangible, negative impact on them. In the case of video cards, it seems so obvious that you'd want to play a game with all the bells and whistles if you can, yet people still insist on using statistics based on the lowest common denominator to make their buying decisions (3DMark FTW!).

Of course, it adds to the confusion (and, to some degree, the entertainment value) when people shift their views suddenly just so that they can continue to complain about something. With the 2900XT, it just seems bizarre that gamers who have previously extolled the virtues of Nvidia cards over and over again over the years are suddenly irritated when someone (like [H]ardOCP) comes to a negative conclusion about the hot, new ATi card.

Ah well, I guess complaining makes Internet forums go 'round...just look at the negative tone of this post for evidence. :D
 
Wow. Always sad to see somebody take quantity over quality


The best playable settings just physically takes far far too long to test, Tim would be writing the article for three weeks if that was the case.
Click to expand...


That's about as disappointing quote as I've ever see. Really says a lot about their intentions. What I see is: "We don't care how good the review is, as long as we get it out fast"


I'll stick with the [H]ard way. :D
 
Bit-Tech has been a favorite of mine, second only to [H]ardOCP, for some time now. I must say, I enjoy their reviews and find them very informative. I personally can't blame them for changing to try to lessen the amount of work that goes into benchmarks. You know, for some gamers, such as people like myself that play mostly online shooters, FPS is the most important thing. Eye-candy is great, but I've found in some games I turn down some of the effects and stuff because it can be distracting and actually hinder you performance. Naturally, when I play single-player games, I crank up everything that I can get away with. I know I am kind of a noob here, and I don't like to make any accusations that I can't back up, but I just find it kind of suspicious that the 2900XT is getting slammed so relentlessly here. Other sites are acknowledging its flaws, such as power requirements and whatnot, but they show it standing up to the GTS a lot better than the overall impression that I got from [H]'s testing. I just read Bit-Tech's review this morning, and was telling my friend about how the 2900XT seemed to fair a lot better in their benchmarks (aside from the power pull and driver instability, of course). Then I pop in over here and find an article discrediting their method. Just seemed kinda funny to me. Once again, I hate to point a finger, but I am just suspicious of what seems to be "R600 bashing at all cost". Just my two cents. I personally will be waiting for an actual DX10 game to be released before I fork out the dough for any of these DX10 cards. I'll just see what looks best TO ME at that point. No disrespect intended. I will continue to enjoy both [H]ardOCP and Bit-Tech.
 
The fact of the matter is almost any video card will give you stellar framerates now days if you turn down the eye candy. That is not a hurdle any more. It is all about single player experience. Then again, playing CS:S at 2560 resolution with all the eye candy turned on is a badass experience as well. :D
 
No, I am not saying that I enjoy playing at 800x600 with minimum everything, just that in online FPS's, frame rates matter. My point was simply that their method would be fine for some people. I can see the benefit in the way you guys do your testing. I am just a conspiracy-theory kind of guy, and thought it strange that other sites show the 2900XT performing better, thats all. Yes, I know, "Of course it performs better, with all the settings turned down," But I would think that if a card can do 100fps @1600x1200 with whatever settings, (just an example), then I can be confident that it will give me decent frame rates with the eye-candy turned up. Once again, just my $.02.
 
Since you guys apperantly dont use timedemo's how to you keep the tests similar when testing cards? Do you run thru a room shooting and try to mimic that same movement for a different video card? Or am i way off here?
 
Deejay3d said:
Then I pop in over here and find an article discrediting their method. Just seemed kinda funny to me. Once again, I hate to point a finger, but I am just suspicious of what seems to be "R600 bashing at all cost".
Click to expand...

Don't confuse "discrediting their method" with HardOCP being critical of THE method overall. I read through many site reviews of the 2900 this week that used terrible methods and terrible games and/or benchmarks. My comments regarding the method were in no way shape or form focused on BT's work.

And there is no bashing here. I think Brent and I are firm in our throughts that there is just absolutely no reason to buy a 2900 at this time considering other alternatives. If telling our readers that is "bashing," then we shall bash on! :D
 
forcefed said:
Since you guys apperantly dont use timedemo's how to you keep the tests similar when testing cards? Do you run thru a room shooting and try to mimic that same movement for a different video card? Or am i way off here?
Click to expand...

Exactly, through practice and experience we play through a game the same way each time, making sure to turn at the same spots, stand at the same spots, perform the same actions at the same times in the same ways. Due to the AI and randomness this is not 100% perfect each time, but it does 100% represent how a gamer plays the game, it will never be 100% exact for a gamer each time either. Many games are more random, such as BF 2142, but it does represent real-world gaming.

Keep in mind that this done for the framerate over time graphs, to backup the settings we found playable. To figure out what we find playable we actually play the games through, several levels, several scenarios, to try and see what the best balance of performance and IQ is, this is where we determine the highest playable settings, through actually sitting there for hours playing the games. The graphs simply backup our analysis.
 
I didn't really like the new way H started to test mainly because it was different. It forced me to think about things in a different way. I then took a minute and realized that I liked this new way, and it makes enough logical sense. I think the problem that everyone has is the way they describe the comparison between the 2.

In other words, people tend to think Real-World Testing > Canned Benchmarks, when really it's Real-World Testing and/or Canned Benchmarks. It's a different method of gathering data and if you process the data enough from both methods you can gather the same conclusions that the other method has. It's just that Real-World Testing tends to give us an easier path to that conclusion: Will The Games Installed on My Computer Run Better?

oh well.. I never really visited their site unless H posted it among others when something big drops.
 
A great example to why we need Testing done with filtering effects turned on is in the H HD2900XT review. When they compare AF filtering, you see the difference between no AF and 16AF???? On both cards its an amazing difference. So I wanna know how much AA./AF I can run with my games now. I'm a $300 gamer, i refuse to spend more than $300 on a any videocard. Therefore yes, I paid my money for some AA/AF!
 
I liked Bit Tech; I will likely still visit their site altho not palce as much emphasis.

Re: style of benchmarking, I prefer the [H] way over 3dmark and standardized time demos; its just too easy to cheat on those and both NV and ATI ahve proven they are willing to cheat if they can find a way to do so.
 
XamediX said:
A great example to why we need Testing done with filtering effects turned on is in the H HD2900XT review. When they compare AF filtering, you see the difference between no AF and 16AF???? On both cards its an amazing difference. So I wanna know how much AA./AF I can run with my games now. I'm a $300 gamer, i refuse to spend more than $300 on a any videocard. Therefore yes, I paid my money for some AA/AF!
Click to expand...

Exactly, AF makes a very large impact in the gaming experience, that impact increases at higher resolutions and higher texture resolutions. IMO AF should be a given now, it should IMO be enabled in every game by default at the highest level, there is no reason to have to game with subpar texture filtering in today's world of gaming.
 
Apples to apples is by far a better way to review a card if what you want is performance numbers and comparison.

Apples to oranges would be fine if I was looking at buying that card, but other than that they don't tell my anything at all for comparison.
 
Deejay3d said:
But I would think that if a card can do 100fps @1600x1200 with whatever settings, (just an example), then I can be confident that it will give me decent frame rates with the eye-candy turned up. Once again, just my $.02.
Click to expand...
Actually, I think the [H]ard reviews demonstrate pretty consistently that this just isn't true. For example, turning up the "fancy grass" in Oblivion is apparently still making the ATi cards crawl while the Nvidia cards run. Now, you still have the option to ignore that factor (and that particular issue would certainly be low-priority for me if I was building a new rig) but the fact is that the issue is still there and you'll get an improved visual experience with one of the high-end Nvidia cards than you will with the ATi in that game.

Providing information is what all those pages before the conclusion of a {H]ard review seems to be about. If you don't care about the difference between 2x AA and 4x AA, or the difference between grass "high" and grass "medium," or whatever other differences get exposed in the review, you're free to ignore it. The thing is, though, if that information isn't even present because a reviewer goes the LCD (sorry, lowest common denominator ;)) route then you're not going to know until you buy the card and see it for yourself. If that happens, and it turns out that you do care about the difference in graphical fidelity, then whatever review you relied upon didn't give you the information you needed.

As the guys so often point out, there's no shortage of "apples-to-apples" reviews out there if all you're looking for are slammin' frame rates. The fact that there are reviews here that include a great deal of additional information can only be a good thing, and losing a member of that small club can only be a bad thing.
 
Who gives a shit? This is just another chance for [H] to polish their ego.
 
Steve said:
I'll second that.
Click to expand...

I'll third that. I enjoy their editorials a lot as well.

kent said:
Who gives a shit? This is just another chance for [H] to polish their ego.
Click to expand...

I personally cannot understand how you look at this as ego polishing in any way. We all have a certain belief around here as to what is more relevent to our readers in terms of reviews.

If you would like to know how card xyz does in 3DMark there are many websites out there that can cater to your liking.
 
kent said:
Who gives a shit? This is just another chance for [H] to polish their ego.
Click to expand...

Maybe, but I do like how [H] pushes the cards to the limit. They should do that. That's testing.

Running preprgrammed instructions time and again is not dynamic and not very useful. Good job [H], I'll be sticking with your reviews.
 
all the more reason why hardocp has been number one on my reading list. your reviews play integral role on how i spend my upgrades money. keep up the good work! i'll still go back to bit-tech for case mods articles, but i'll always go to hardocp for the latest and greatest in hardware news.
 
The only issue I've had with [H]'s review method, and I can see why someone like BT would return to 'apples to apples' because of it, is that it's opinion.

"Maximum playable setting" means different things to different people. And while I generally agree with [H] on the FPS number to shoot for, when it comes time to 'kick the settings up', WHICH setting to kick is a big variance. [H] will invariably kick up the resolution before FSAA or in-game settings. Me, I'd much rather game at 1024x768 with 4xAA and all in-game settings maxed out than 1280x1024 with no AA.
 
dderidex said:
The only issue I've had with [H]'s review method, and I can see why someone like BT would return to 'apples to apples' because of it, is that it's opinion.

"Maximum playable setting" means different things to different people. And while I generally agree with [H] on the FPS number to shoot for, when it comes time to 'kick the settings up', WHICH setting to kick is a big variance. [H] will invariably kick up the resolution before FSAA or in-game settings. Me, I'd much rather game at 1024x768 with 4xAA and all in-game settings maxed out than 1280x1024 with no AA.
Click to expand...

But that's almost negated by the FPS scores they provide: AVG, MAX, MIN. You can see for yourself where you fall on that line, below or above.
 
i've been with the [H] since God knows when (my join date here at the forum says 2001 but i've been here much, much longer than that) and i have never, EVER, found a need to go to any other site.

this thread is a testament to why

i don't know, call me crazy but it makes incredible sense to me to review cards as they are used by real gamers, including myself. mind you i don't get paid by anyone to review video cards, cater to a readership, or to even test video games - i am an honest to God lifelong gamer and love the hobby with all my heart. thus, i am able to see past the marketing and controversy when these kinds of issues arise.

look, FPS benchmarks were great and all when we were all kids playing Quake on our brand new Voodoo; and FPS was an actual measure of immersion as it was what determined if the game was even playable or not.

but video card technology evolved, and so did industry.

in the age of 22"+ widescreens, HDTVs, and 1600x1200 being an "average" resolution on all mainstream video cards, the FPS equation does not give us a proper result any longer. Kyle is right when he says that features such as AA are what really matter, and to shut off such features to test "raw horsepower" is simply buying into the way in which it benefits only the hardware companies (and now, the hardware sites and their readership, also known as the advertising money)

shame on you BitTech, you are now gone from my bookmarks

and as for the whole "[H]ardOCP has an ego" thing: don't confuse pride with standing up for the truth, which is what i think Kyle realized back when the site switched over to real-world testing. and when they got sued by Infinium Labs.

but hey if you don't like outspoken people, and trust me... it's not called the [H]ardOCP (and the [H]ardForum) for nothing - then just LEAVE!

i'm sure [H]ardOCP understands that they loose a good chunk of readership because of how they choose to run their company, and that is something to be admired today in a world full of big Internet ad money, "guerrilla" marketing, and hardware and gaming companies thinking they can spin and then piss all over the people who turned them from the joke of the electronics and entertainment industries to the international powerhouses they are today.

and that reminds me, to NVIDIA (regarding the Lost Planet demo): slick move, real slick pulling one over not just AMD/ATI, but over all gamers alike. and i wasn't even talking about the 8800 Ultra

some things never change, huh?

To Kyle, Brent, Steve, and everyone else, including the readers, who make this place rock: don't ever change

i don't want to resort to reading Newegg customer reviews
 
Rich Tate said:
I personally cannot understand how you look at this as ego polishing in any way. We all have a certain belief around here as to what is more relevent to our readers in terms of reviews.

If you would like to know how card xyz does in 3DMark there are many websites out there that can cater to your liking.
Click to expand...

Amen brudda!

dderidex said:
The only issue I've had with [H]'s review method, and I can see why someone like BT would return to 'apples to apples' because of it, is that it's opinion.

"Maximum playable setting" means different things to different people. And while I generally agree with [H] on the FPS number to shoot for, when it comes time to 'kick the settings up', WHICH setting to kick is a big variance. [H] will invariably kick up the resolution before FSAA or in-game settings. Me, I'd much rather game at 1024x768 with 4xAA and all in-game settings maxed out than 1280x1024 with no AA.
Click to expand...

Not all of us are going to have the latest and greatest hardware to play with. As such, each of us has to take the results [H], and other hardware sites, have obtained and extrapolate through experimentation to find the the resolution and detail level that meets our individual satisfaction.
 
dderidex said:
The only issue I've had with [H]'s review method, and I can see why someone like BT would return to 'apples to apples' because of it, is that it's opinion.

"Maximum playable setting" means different things to different people. And while I generally agree with [H] on the FPS number to shoot for, when it comes time to 'kick the settings up', WHICH setting to kick is a big variance. [H] will invariably kick up the resolution before FSAA or in-game settings. Me, I'd much rather game at 1024x768 with 4xAA and all in-game settings maxed out than 1280x1024 with no AA.
Click to expand...

Actually, we strive to enable in-game settings as #1, the way I feel about it is that the game developer has a certain environment/mood/feel for a game that they want you to experience, unless you are running the game at the highest settings you are not going to get the general feel the game is suppose to have. So I aim to enable in-game options to their highest levels first.

As for when to enable a higher resolution or AA, that completely depends on the game. For example in Doom3/Quake 4 it is better to raise the resolution than the AA, the experience is better with a higher resolution instead of AA because the game is so dark AA makes little difference. Then there are games like FS X that benefit a lot more from enabling AA instead of a high resolution, there is so much aliasing going on we set 2X as a minimum AA setting that should be enabled at all times and adjust the resolution accordingly. So it completely depends on the game.

Just the fact that Card A was able to play at a higher resolution/AA/in-game settings from Card B alone tells you Card A is faster. Depending on the difference in settings it will tell you if it was faster by a lot, or a little. You don't need framerates to tell you that, it is all about the experience delivered, and higher settings = better experience typically.
 
The video card market is not about frames per second any more; it is about smooth gameplay and immersive digital environments.
Click to expand...

That quote really hit home for me. I agree with it completely and is one of the many reasons why I read the [H]. :cool:
 
Terminus said:
i've been with the [H] since God knows when (my join date here at the forum says 2001 but i've been here much, much longer than that) and i have never, EVER, found a need to go to any other site./QUOTE]

I second that. I didn't join the forum way back when, but the [H] has been my homepage now for many many years. Don't have any reason to go looking elsewhere. If its important they post something about it and if it is worth reviewing they will slice and dice it to pieces and tell you how it crumbled.
Click to expand...
 
Deejay3d said:
No, I am not saying that I enjoy playing at 800x600 with minimum everything, just that in online FPS's, frame rates matter. My point was simply that their method would be fine for some people. I can see the benefit in the way you guys do your testing. I am just a conspiracy-theory kind of guy, and thought it strange that other sites show the 2900XT performing better, thats all. Yes, I know, "Of course it performs better, with all the settings turned down," But I would think that if a card can do 100fps @1600x1200 with whatever settings, (just an example), then I can be confident that it will give me decent frame rates with the eye-candy turned up. Once again, just my $.02.
Click to expand...

I flat out asked bindibagi on there forums, why it was dropped, and replied by saying they couldnt be bothered as it was too expensive. i think there is every right to discredit what they do now... I hate to stir up trouble, but that is just dishonest if i've ever seen it. it's a dis service to the readers over there, and it seriously irks me.

There previous reviews were my only source of backup towards [H]'s results to reassure me as to what i felt, and they dropped them because they couldn't be bothered. its disturbing...


Thank you Kyle, for not being a sell out. *slautes*
 
chrisf6969 said:
I'm in both camps.

I NEED my apples to apples comparison so I can really quantify just how much better a card is at a given setting. That is where I make my decision on buying a card. B/c I still play most of my games at 1152x864 with as high of AA/AF I can do and still get good frame rates.

That said, the apples to oranges comparisons are growing on me. But I prefer when the in game settings are left the same (or VERY close). And only resolution and AA/AF are adjusted. If a lot of in game setting are changed its just too many variables to compare. Its good to see whats the max setting you can use for a given card/game.
Click to expand...

the sites that post apples to apples results never indicate whether or not they turn on transparency aa. this feature exacts a heavy toll on video cards and stands as a true test of a cards potential. why would anyone spend hundreds of dollars on a video card and not flip on as many switches as possible?
 
rennyf77 said:
the sites that post apples to apples results never indicate whether or not they turn on transparency aa. this feature exacts a heavy toll on video cards and stands as a true test of a cards potential. why would anyone spend hundreds of dollars on a video card and not flip on as many switches as possible?
Click to expand...


Brent is head and shoulders above the rest when it really comes to explaining what settings are being used and WHY they are being used. No doubt many of these things are above the average Intarweb readers' heads, but that is not who we want to cater to. We want to cater to folks that want to truly understand what the technology hidden behind the driver sliders is really good for...or not good for.

You are the guys Joe Sixpack asks when it is time to buy hardware anyway, so you are the ones that need to know. We can just hope that Joe is lucky enough to be talking to an [H] reader. :D Yeah, there is a certain amount of pride around here as we are proud of our accomplishments, but without our readers, well....we ain't shit.
 
sad to see em go
but glad to know
[H]ard to the O
is here for us bros
but don't forget those hoes, bros
who know how it goes

fo sho.

*cue coffee-house bassline*


/waits for gunshots

:p
 
You must log in or register to reply here.
Back
Top