XP vs. Vista - A Tale of Framerates

I think all you complaining about the drivers and lack of other cards in the article have never written one yourself and are, therefor, not aware of the sheer amount of work and time involved :p

If I were a part-time evaluator (which most at [H] are) I certainly would not have "expensed" a video card for reviews that will probably just be irrelevant in a few months. Most of the cards review sites use are evaluation cards provided by manufacturers. I think the real reason behind not having any ATI cards is simply the amount of work required for the article would have been more than someone who doesn't do this full time could have handled.

I was kind of annoyed that Kyle and Brent let him get away with regular old benchmarks instead of an [H] performance evaluation though. I, for one, can run double the AA with Vista vs XP in cnc3 without my framerate changing (3ghz c2d, 2gigs pc1066) and on other games I can't get AA to work at all. Things like that would have been brought to the surface using normal [H] evaluations but probably would have required one of the video card reviewers to do it and several (or Brent himself) to get it done in a timely manner.

Just my 2cents.
 
from Article said:
We experienced a strange event while launching this title. For a game that was designed with Vista in mind, it surprised us that the UI had to change from Aero to “Vista Basic” for the game to run. None of our other titles required this. Otherwise, the game installed and played perfectly fine.
Click to expand...

I thought Vista was supposed to turn off Aero for every game whether it was designed for Vista or not? Or am I misunderstanding what "Vista Basic" is?
 
kheldorin said:
I thought Vista was supposed to turn off Aero for every game whether it was designed for Vista or not? Or am I misunderstanding what "Vista Basic" is?
Click to expand...

Aero doesn't turn off on my machine while I'm in game, I keep a browser and IRC client up on my other monitor at all times and they never change.
 
wiper said:
I think all you complaining about the drivers and lack of other cards in the article have never written one yourself and are, therefor, not aware of the sheer amount of work and time involved :p
Click to expand...

Then, simply don't make the conclusion that the problem lies with Vista and not the drivers. Don't get me wrong, the article was great and very informative but it still does not have enough data to rule out that the drivers are not the factor. As far as I'm concerned, both could be a factor but the drivers are the probably the most likely given Nvidia's track record so far.
 
CaptNumbNutz said:
You only had Nvidia Cards available when you wrote that article? An extremely popular and well managed hardware review site did not have any spare ATI cards sitting around for your other reviews?:confused:

So where did all the ATI cards you have been using for past reviews go to? You had an Asus card in review during March, or did you guys have to return that one?
Click to expand...

Jason's located in the Austin office - the video card testers are in a different office (Houston, I believe.)
 
Thanks for the article it was a good read.

This reminds me when XP first came out and 98 SE was the gaming OS of choice at that time 98 SE blew the pants off XP where FPS was concerned, however once XP and drivers matured XP is the gaming OS to beat. Vista will come out on top once hardware makers and game developers start creating games specifically for VISTA with only compatibility of XP in mind. The question is how long will this take?




As for Aero I would guess the issue with Flight Sim X is AERO shuts off/crashes until the game is closed/not running as a process and then Aero restarts it self.
On most games AERO will disable its self when in game and as soon as you minimize or close out of the game its back on that is why turning off AERO or disabling any of its features for gaming will NOT yield any noticeable gain in performance.
 
Good article!

I notice a much large difference between XP/Vista on your 7xxx series video card than the 8xxx series card. It's no secret that NVIDIA has been pumping out more frequent driver revisions for the 8xxx series cards.

This probably explains one of the differences of opinion we see around this forum! Most people that claim similar/better performance in Vista seem to be using an 8xxx series card. Hopefully we see those gaps narrow even more as drivers mature.

Another thought about WoW: Remember the Killer NIC? The only game that seemed to benefit significantly was WoW - perhaps this is due to the large amounts of network or disk I/O this particular title uses/requires?



Catweazle said:
Then there's the concept of "level playing field" to confront. A fresh, clean install of Vista is no longer necessarily the 'optimal state' as it was with XP. Wipe a hard drive, bung Vista and a game on it, and then run a few tests and you aren't necessarily getting a true indication of performance. Technologies not included in XP mean that it'll gain performance after it's been in use for a little while, and to date I've seen no hardware site accomodate that concern when conducting tests and producing comparitive reports.
Click to expand...


Very good point - a fresh install of Vista will perform significantly worse than a several-day-old install of Vista: Superfetch, Indexing services, etc.

Kyle: Were your benchmarks performed in-order? i.e., was WoW the last game you benchmarked? It would be interesting to go back and benchmark the first title you did at the end of your run and see if you get the same results...



Jason_Wall said:
It's my fault that the driver revisions weren't specifically mentioned - sorry about that.

NVIDIA just released the 158's last week, which was AFTER we completed and locked this article. Bad timing, what can I say.

All of the drivers used were WHQL certified, and were the most recent drivers available (up to, but not including the 158's). For the 7600GT on XP, it was the 93.71's. For the 8800 GTS on XP, it was the 97.92's. For the both cards in Vista, we used the 100.65's.
Click to expand...


We DEFINITELY need to see a refresh on this article with the 158 drivers :) Those are the drivers that finally bumped my CS:S framerates up to where they were in XP :)
 
Eh heh.. there is a funny pattern emerging on these forums:

Test shows Vista is clearly slower than XP on games.

Post 1: Denial
Post 2: Excuse
Post 3: Circumvent
Post 4: Blatant lie
Post 5: Teehee told you so!

And flush from the start.

[TheRock mode ON] It doesn't matter if it's the 'drivers at fault', Vista is slower than XP[TheRock mode OFF]
 
I know Jason's been working on this one article for at least the amount of time I've been working on evaluating OSes, which themselves are massive. All of [H]'s tests are time intensive, but this one was particularly time intensive because Jason kept having to retest on two different OSes, and each time a driver came out he'd have to restart every test, every time a patch came out he'd have to restart that game's test... etc.

Yes, an ATI card would have made a good test, but it would have complicated issues dramatically and extended this article from a "less timely" status to an "irrelevant" status.

So, don't knock him too hard for testing 8 different games on 2 OSes with 2 video cards. That's 32 different combinations. Three video cards would have made it 48 different combinations.

That's more than anyone else does, and for good reason - game testing sounds cool but ultimately it can be very, very tedious and time consuming.

Now, here's the call to action: As far as I've worked for them, HardOCP is a big believer in the "Stand Back, I'm Going To Try Science!"-tific method. If you believe that there's a gap in the research, please feel free to fill it with your own experiments. Even if you think HardOCP is dead on, I'd love to see someone else replicating the results! :D
 
Just as last time when a Vista article appeared on the [H]:

Incomplete and insufficient testing methods make this article half true and nothing but a bad example of how NOT to test. 40% (or whatever the amount of ATI users are) of the user base have been completely ignored by this article (once again). Is [H] turning into [h] or even ?
 
calebb said:
Another thought about WoW: Remember the Killer NIC? The only game that seemed to benefit significantly was WoW - perhaps this is due to the large amounts of network or disk I/O this particular title uses/requires?
Click to expand...

Entirely possible - especially because Vista rewrote the entire TCP/IP stack, which causes it to ramp up faster and slow down slower. I could get into networking here, but basically, TCP sends little packets first, then bigger and bigger packets until it finally finds the packet size that works. If the packet goes through, it sends a bigger packet next time. If the packet fails, it sends a smaller packet or if enough failures occur, it restarts the count with the smallest packet and works it's way back up. Windows Vista doesn't change this too much but it does send bigger packets earlier in the cycle and doesn't reset back to the smallest packet size quite as quickly.

We covered this at Network Performance Daily to some extent.
 
ScYcS said:
Just as last time when a Vista article appeared on the [H]:

Incomplete and insufficient testing methods make this article half true and nothing but a bad example of how NOT to test. 40% (or whatever the amount of ATI users are) of the user base have been completely ignored by this article (once again). Is [H] turning into [h] or even ?
Click to expand...


Well they already explained tous what happened to ATi so i won't say anything but how would you recommend you compare the performance of an Operating System.

Seems to me if I wanted to compare performance of two Operating Systems I take one machine install both Operating Systems and run the same benchmark in both Operating Systems and then compare the framerates. This way the only variable is the Operating System.
 
stabilep said:
Well they already explained tous what happened to ATi
Click to expand...


Explaining doesn't make it better. They should have used one nVidia card and one ATI card at the least. If the article takes longer to finish because of this, then so be it. Better release a finished article than to neglect half of the people that read the article.
 
It's just like when XP came out and 2K was better for gaming... it'll get ironed out.
 
This review has one major flaw: You're only using video cards from the manufacturer whose track record in Vista has been absolutely abysmal! Why not throw an ATI card or two into the review and see how they fare? I run an X1900XT and while I see slightly lower framerates in Vista, they're nowhere near the discrepancies you show in your testing.

Another thing I'd like you to talk about would be the smoothness of the gameplay. Vista may be a bit slower in the overall framerate catagory, but what about the stability of that framerate?
 
ScYcS said:
Just as last time when a Vista article appeared on the [H]:

Incomplete and insufficient testing methods make this article half true and nothing but a bad example of how NOT to test. 40% (or whatever the amount of ATI users are) of the user base have been completely ignored by this article (once again). Is [H] turning into [h] or even ?
Click to expand...


Yep… along with the same type sweeping silly conclusions. We get testing that does not use video cards that have the best Vista drivers and then get sweeping foolishness like “In the end, it looks like the new, bulky, poorly supported operating system is at the root of it all.” And then when called on it get a ridiculous excuse that ATI hardware was unavailable.

Hey here is a novel idea… if you don’t have the hardware needed to give a valid and fair look at Vista GET IT before doing the article and don’t do things half assed.

It appears as if the testing was set up to generate results that would reinforce the ridiculous conclusions jumped to in “30 Days with Vista”.It appears the authors got the results they wanted because the testing was designed to give the results they wanted... pretty sad.
 
Finn said:
Eh heh.. there is a funny pattern emerging on these forums:

Test shows Vista is clearly slower than XP on games.

Post 1: Denial
Post 2: Excuse
Post 3: Circumvent
Post 4: Blatant lie
Post 5: Teehee told you so!

And flush from the start.

[TheRock mode ON] It doesn't matter if it's the 'drivers at fault', Vista is slower than XP[TheRock mode OFF]
Click to expand...

congrats on offering something to the discussion other than trolling. :rolleyes:
 
Brian Boyko said:
I know Jason's been working on this one article for at least the amount of time I've been working on evaluating OSes, which themselves are massive. All of [H]'s tests are time intensive, but this one was particularly time intensive because Jason kept having to retest on two different OSes, and each time a driver came out he'd have to restart every test, every time a patch came out he'd have to restart that game's test... etc.

Yes, an ATI card would have made a good test, but it would have complicated issues dramatically and extended this article from a "less timely" status to an "irrelevant" status.

So, don't knock him too hard for testing 8 different games on 2 OSes with 2 video cards. That's 32 different combinations. Three video cards would have made it 48 different combinations.

That's more than anyone else does, and for good reason - game testing sounds cool but ultimately it can be very, very tedious and time consuming.

Now, here's the call to action: As far as I've worked for them, HardOCP is a big believer in the "Stand Back, I'm Going To Try Science!"-tific method. If you believe that there's a gap in the research, please feel free to fill it with your own experiments. Even if you think HardOCP is dead on, I'd love to see someone else replicating the results! :D
Click to expand...

Well put - and I agree for the most part. While it would've been nice to do more, only so much can be done with finite resources. Driver revisions complicate things and at some point you just have to cut things off if you ever want to publish. That's all understandable (to me). However, I also feel strongly that it would've been better to be more honest/upfront (either in the story title or in the intro blurb on the front page) that this was an nVidia only test, just to set reader expectations.

As far as what other people have done...there's a decent "mainstream" media comparison (http://www.extremetech.com/article2/0,1558,2096940,00.asp) that used the initial Vista drivers available (so it is out-dated), but did cover at least one card from each camp.

Maybe in the future, if a similar type evaluation is done, it might make more sense to use fewer games to make it more feasible to cover both (or however many there may be) camps?
 
I think the title update would probably be the best thing for the article, to be honest.
 
I'm just curious why, for CPU and Video Card benchmarks, the [H] generally jumps up and down and screams that nothing but real-world benchmarks conducted to determine max playable settings are meaningful, but for this article, it's simple framerates?

It's like the C2D vs A64 article that had everyone up in arms. The conclusion was "C2D is faster, but it doesn't matter in current games." I'm curious if a similar testing methodology would have yielded "XP is faster, but for current games it doesn't matter."
 
Brian Boyko said:
Yes, an ATI card would have made a good test, but it would have complicated issues dramatically and extended this article from a "less timely" status to an "irrelevant" status.
Click to expand...

Well, IMO (and that's just my opinion, nothing more), your article IS already "irrelevant" (to me) even without ATI cards since it doesn't use the more recent drivers for the cards that were tested... And FYI, 158.18 Vista drivers were available for downloading at an official Nvidia's page since at least 4/17 (they didn't have "whql" thingie until 05/02 but I don't think it is relevant at all and has any impact on performance or stability).
 
darkangel74 said:
He does have a point.:)
Click to expand...

ORLY?

its a known fact the Vista NV drivers they used are considerably shittier than the 158 set that is out now. i GUARANTEE they'd see a closer gap in performance if they used the new set and would put money on it. driver maturity = better framerates = simple concept
 
LstOfTheBrunnenG said:
I'm just curious why, for CPU and Video Card benchmarks, the [H] generally jumps up and down and screams that nothing but real-world benchmarks conducted to determine max playable settings are meaningful, but for this article, it's simple framerates?

It's like the C2D vs A64 article that had everyone up in arms. The conclusion was "C2D is faster, but it doesn't matter in current games." I'm curious if a similar testing methodology would have yielded "XP is faster, but for current games it doesn't matter."
Click to expand...

They didn't ignore that completely. They did look at image quality at the various settings used in the test, and at identical settings, the image quality was the same, so in their opinion (and they took screenshots for us to disagree/agree) framerates were the only variable in the gaming experience. They had to limit the number of variables to do a true "XP vs. Vista" comparison. It'd be different if this were a "Vista gaming experience" article...
 
LstOfTheBrunnenG said:
" I'm curious if a similar testing methodology would have yielded "XP is faster, but for current games it doesn't matter."
Click to expand...

That has been exactly my experience with Vista verse XP and gaming… there may be a difference in frame rates but it doesn't matter because you cannot tell the difference when playing games using either OS.

But my guess is a conclusion like that will not generate enough buzz or be negative enough to get another Slashdot notice.
 
IrishRangerWingMedic said:
How long after install of the games did you test the rates? It takes some time (?1-2 weeks) for Vista to learn what applications you use regulary and cache them for superfetch.
Click to expand...
Superfetch affects how long it takes for data to get from the HDD to the memory, not how fast your GPU can push pixels.
 
LstOfTheBrunnenG said:
Superfetch affects how long it takes for data to get from the HDD to the memory, not how fast your GPU can push pixels.
Click to expand...

100% true.

However, superfetch will rearrange your hard drive to put frequently used applications in the 'fastest' locations on your hard drive (outer edge from what I understand) - even while you are playing games... (low priority I/O so it won't affect it as much as the same operations would in XP... but there are still background tasks going on in the days/week after you first install Vista).
 
LstOfTheBrunnenG said:
Superfetch affects how long it takes for data to get from the HDD to the memory, not how fast your GPU can push pixels.
Click to expand...

Yes, but if that (or other processes) is accessing the disk in the background, it could affect how quickly data gets to the GPU so that it can push pixels. Not sure if that is an issue, or how much it would have an impact, but it'd be nice to rule it out...
 
Why, oh why, oh WHY did you use an NVIDIA card?! :eek: Everyone knows that NVIDIA + Vista = flaming pile of crap right now.

I can say from personal experience that if you had used an ATI card, that you would have seen zero difference between the two.

Good article, good intentions, but it was a poor choice of hardware, and I do have to criticize you for your choice of graphics card; it's a known fact that ATI is better in Vista right now, and for a website as prestigious as [H]ardOCP to make such a bad choice of hardware is baffling.
 
InorganicMatter said:
Why, oh why, oh WHY did you use an NVIDIA card?! :eek: Everyone knows that NVIDIA + Vista = flaming pile of crap right now.

I can say from personal experience that if you had used an ATI card, that you would have seen zero difference between the two.

Good article, good intentions, but it was a poor choice of hardware, and I do have to criticize you for your choice of graphics card; it's a known fact that ATI is better in Vista right now, and for a website as prestigious as [H]ardOCP to make such a bad choice of hardware is baffling.
Click to expand...

Not necessarily - when you step up to the 158.xx drivers, the performance gap narrows quite a bit! I wouldn't call it a flaming pile of crap, but we'll need to wait for someone with hours & hours to spend carrying out the long arduous task of repeating all these tests with yet another nvidia driver release :)

Also, I don't think there is a 'zero' gap for ATi cards, even though their drivers are much more mature and feature-complete.
 
When I recently bought the parts to build a new system, I stayed with XP Pro, instead of going with Vista. Bad performance was just one of the MANY reasons I am staying away from it. I jumped on XP when it released but Micrsoft screwed up on a LOT of things in Vista.
 
Kyle - I think it's pretty easy to still point the finger at graphics drivers. In your comparison, you used two Nvidia cards. Nvidia seems to be suffering from driver problems in Vista. Why not test using some ATI cards and see if you get similar results?
 
I don't know if you've noticed boss, but when the game goes to disk during game play - whoch it always has to, frame rates plummet.

Frame rates are multifactorial - GPU frequency, pipelines, memory; Sytem RAM, Bus speed; Processor frequency, FSB; Hard-disk RPM, Cache, data transfer width....etc


LstOfTheBrunnenG said:
Superfetch affects how long it takes for data to get from the HDD to the memory, not how fast your GPU can push pixels.
Click to expand...
 
My 8800 is working great - Not one problem! Have you had problems with Nvidia drivers?

InorganicMatter said:
Why, oh why, oh WHY did you use an NVIDIA card?! :eek: Everyone knows that NVIDIA + Vista = flaming pile of crap right now.

I can say from personal experience that if you had used an ATI card, that you would have seen zero difference between the two.

Good article, good intentions, but it was a poor choice of hardware, and I do have to criticize you for your choice of graphics card; it's a known fact that ATI is better in Vista right now, and for a website as prestigious as [H]ardOCP to make such a bad choice of hardware is baffling.
Click to expand...
 
What were the visual settings and the like for Vista (the OS itself)? Like animated cursors, desktop effects, window effects, sidebar, etc. They can effect performance and if you had some more of them turned on than XP, it probably effected the results a little from memory usage.
 
InorganicMatter said:
Why, oh why, oh WHY did you use an NVIDIA card?! :eek: Everyone knows that NVIDIA + Vista = flaming pile of crap right now.

I can say from personal experience that if you had used an ATI card, that you would have seen zero difference between the two.

Good article, good intentions, but it was a poor choice of hardware, and I do have to criticize you for your choice of graphics card; it's a known fact that ATI is better in Vista right now, and for a website as prestigious as [H]ardOCP to make such a bad choice of hardware is baffling.
Click to expand...

I wouldn't say choosing nVidia was a "poor" or "bad" choice - they still are in the lead in the discrete graphics market (according to the Q4 reports) and are the only manufacturer with a DX10 card publicly available.
 
markt435 said:
ORLY?

its a known fact the Vista NV drivers they used are considerably shittier than the 158 set that is out now. i GUARANTEE they'd see a closer gap in performance if they used the new set and would put money on it. driver maturity = better framerates = simple concept
Click to expand...


Post 5: Teehee told you so!

yeap he's got a point :)
 
Jason_Wall said:
We would have loved to try the 158's. Unfortunately, they came out just as we were putting the finishing touches on the article. We'll definitely be interested to take a look at them when we work on a follow-up to this article.
Click to expand...

Definately a follow-up is in order. Speaking purely from the standpoint of a G80 board owner, the 1xx.xx series drivers for Vista when compared to the 9x.xx are like night and day in some situations.

Not to downplay the article because I think it's awesome, but the first thing I thought just now when I read that you guys were using the 97.92 for the G80 was "Well no duh, those drivers we're awful!". :)
 
InorganicMatter said:
Why, oh why, oh WHY did you use an NVIDIA card?! :eek: Everyone knows that NVIDIA + Vista = flaming pile of crap right now.
Click to expand...

From my perspective that seems like quite an overstatement... but you know what they say about opinions and a.... ern.... ahem.... well... let's just say everybody has one.

Since the 150 series drivers came out things have been quite rosy for the nVidia camp. There are still a few hiccups here and there but nVidia's driver development team really seems to be putting their noses to the grindstone.

It would have been nice to see an ATI card in the bunch though.
 
Not really surprising, is it?

Back when XP was new and the net was flooded with 98 vs. XP tests, 98 had the upper hand in performence in them all, but today we all use XP and not 98.
 
You must log in or register to reply here.
Back
Top