Looking for some test suggestions [Archive]

Joe Average

06-05-2008, 09:28 AM

I don't use applications like PCMark because they don't provide consistent enough results for me. Run it once, get a baseline, reboot, run it again and expect minimal differences and that's not what I get, so there goes that program. And also it doesn't run on every current version of Windows, also, so that's almost a double negative against it.

What I do is set up situations for applications that reside on a second hard drive that I know I can trust and will provide reliable and consistent performance results. I have a few:

- a directory with just over 13GB of mp3 files in it, all properly tagged and categorized. I use this for testing how fast the media player of the given OS can load every song into it's library. As I've noted in posts in the past, performing that operating using WMP11 on XP 32 bit takes about 13 mins; doing the same exact thing with XP x64 takes just under 3 - same hardware, same processor, same everything except the OS and hardware drives and it's literally 4x faster.

- Quake 3 Arena. It runs on any version of Windows, it sits in a single directory, and provides me with an excellent quick way to test CPU/RAM/Video capability based on the OS and the video drivers primarily. I use a little application called Q3Bench which makes benchmarking very easy, configurable as to number of runs, resolution, etc, and then provides the results in a nice simple text file. As also posted in the past, XP x64 is roughly ~20% faster running this game compared to XP 32 bit on the same exact hardware platform.

- PCWizard (version depends on the latest, right now that's 2008) from the makers of CPU-Z. I use the CPU/RAM/disk benchmarking capabilities to get measurements in each OS.

- Geekbench 32 bit / 64 bit. A few years ago a website produced these tiny command line apps that do specific CPU/RAM testing to show how fast a given hardware platform can crunch the numbers during testing. It runs a series of mathematical functions to check for processing speed and RAM bandwidth to come up with a composite score that's pretty consistent. It's almost impossible to find nowadays as the people that created it have now branched it into a commercial product (with a free limited capability stripped down version) but for me, I prefer the old command line versions I have. Tiny, fast, efficient, and the test only what truly matters to me.

- HDTach for testing hard drive burst and sustained read speeds (averages, of course). HDTune is ok, but I've been using HDTach for a long time now, still reliable, still useful. I do use HDTune also for the basics, including its ability to monitor the S.M.A.R.T. status of a hard drive.

- DVDDecrypter for testing ripping speed of an optical drive, and then once it's on the drive (I use the same DVD always, my retail pressed copy of Rounders) I then crunch it with the highest quality using DVDShrink to test for crunching/processing speed. You'd be surprised what a difference there is doing that between the various versions of Windows available these days. :)

That's about the gist of my testing collection. There really isn't much else out there in terms of "testing software" or benchmarking applications that I have any use for. My belief is the best way to tell how fast a given OS is becomes testing the given OS doing specific things that are easily duplicated across the OSes being tested. Yes, I realize that "testing software" is designed to do that very same thing, but so far, I haven't found one single "all in one" testing application that works across all the versions of Windows I care about testing and doing it consistently.

I had high hopes for PCMark when it first appeared, but with the changes in Vista and the fact that PCMark doesn't really work with it (you can use it with Vista if you know how to tweak some stuff, but it doesn't work very well with 64 bit OSes and doesn't offer any 64 bit testing aspects so that's a third negative, come to think of it), it's just not something I use or recommend any longer.

Nobody has created the coolest most useful all-in-one Windows OS testing suite, hence my use of several different methods of doing the testing manually. I really don't think there's a better way to do it, personally.

Oh, and as always, the most important tool in any tester's kit:

Imaging software. You know I prefer True Image, I know you prefer Ghost. Either one works, and is an absolute must-have for system testing across multiple OSes, period. If you really are serious about doing the testing, the hardware platform must never change, not even swapping hard drives that are the same make, brand, model, cache size, etc. If you want accurate results, you must keep the exact same identical platform and only alter the OS, drivers, and possibly the applications depending on what you're doing. That's the only surefire 100% way to know you're testing the differences between the software - the OSes - and nothing else.

'Nuff typed...