Is there a way to blindfold test Windows sound output methods?

Nazo

2[H]4U
Joined
Apr 2, 2002
Messages
3,672
For quite some time I've felt like something just didn't sound right with my Windows sound. Initially I determined it to be a fault of drivers I was using for the soundcard I had at the time (Asus DG which always had horrible drivers and a third party had offered improved modified drivers that were a bit of a hodgepodge -- to make things worse, simply removing them didn't help at the time, but when I did a clean reinstall of Windows it sounded much better versus before it,) but I'm now using a USB DAC with very minimal drivers and once again I find myself feeling like it just has something missing but not really being able to put my finger on it. Recently I setup a Raspberry Pi with a cheaper third party equivalent to the HiFiBerry DAC+. Suddenly I find that the sound coming out from it is just positively blowing me away for some of the same songs even. Somehow it just sounds clearer or something I can't quite pinpoint. (I do not have golden audiophile ears. I consider myself to be a baby audiophile as sound is terribly important but I don't go all out insane over it. Mostly.) I also tried a Fiio E18K since it can handle my HD650 headphones along with a different RPi and once again the sound just seems somehow not just clearer, but more natural and realistic even.

One thing I've been trying to figure out is there are various Windows sound methods, each of which people say sound different in various ways. But I'm having troubles really pinpointing. I would swear some like DirectSound do seem somehow softer and with almost a hint of a reverb or something, but I really can't even be sure. And how much of this is cognitive bias because I believe there is a possibility there is a difference and that the worse methods should sound worse? I need to test each of the output methods and figure out what is the best and I need to be relatively sure the result is accurate. Perhaps even with this I could better pinpoint whatever it is that seems to make such a difference. Perhaps even there is no difference and once again there is a cognitive bias (though it's odd because the USB DAC on my PC should be of a significantly higher quality than the generic HiFiBerry and bloody well had better be better than the Fiio E18K given the price difference.) I need to at least start narrowing all this down even if it's going to be hard to ever truly pinpoint even if it is simply bias. Another thing I'm trying to pinpoint is if it could somehow be Windows itself. As I said with the Asus DG hodgepodge drivers, uninstalling them did nothing, but when I later reinstalled Windows from scratch it suddenly sounded better. It leads me to wonder: what if it's something that happens over time or wasn't initially installed on the fresh Windows installation which gets installed later on? Anything I can do to help me narrow things down would be nice and this seems like a good place to start since I can't even right now specify exactly what I'm looking for.

Besides DirectSound and of course ASIO, I think I should test stuff like WASAPI though. Obviously Foobar2000 comes to mind as something with blindfold tests, but it tests input files, not output methods. My best guess is I need a console utility that can output via different methods (perhaps a Windows port of mplayer? I've never actually messed with it though and wouldn't have a clue how to make it do this though. As far as I know it doesn't support those more advanced sound output methods anyway) and to setup some sort of batch script. At the very minimum, if I could even manually write some simple script to test this back and forth on the fly instead of manually switching everything around back and forth it might still help some, though something random obviously would be better (well, it needs a way to actually test me though. Eg like how Foobar2000 asks you which is which and then grades your response based on which ones it actually played so you are much better able to know if you're just being biased or not.) I think all of this could be done via a script, but as many batch files as I've done over the years, I've never really messed with some of the stuff that I'd have to do for such a thing. (Mostly simple if statements, errorlevel checks on the old MS-DOS based choice.com, and etc. Very very little of the more advanced stuff the Windows console has added in its NT-based iterations like modern Windows versions.)
 
You are talking about doing ABX testing, which is pretty complicated to do properly for the reasons you have already discovered.

If you do wind up making some sort of scripting tool, I'd be interested in trying it as well as I have a whole bunch of dac/amps that I want to do some real AB testing on by quickly switching the outputs in Windows. Manually doing it is not simple as some music players simply stop working when you change it with Windows, and you need another switcher that all your amps are connected to with one pair of headphones, and you then need to switch that at the same time that you switch the Windows device, so you get an immediate switch. I have run my head around in circles trying to figure out how to do this with off the shelf equipment and come up frustrated.
 
Everything is turned off officially. If anything is currently active it is doing so against the settings.

You are talking about doing ABX testing, which is pretty complicated to do properly for the reasons you have already discovered.
Well, I already know it is. Cognitive bias is a terrible thing. At least I'm aware of it. It's a truly frightening thing when one is not aware.

If you do wind up making some sort of scripting tool, I'd be interested in trying it as well as I have a whole bunch of dac/amps that I want to do some real AB testing on by quickly switching the outputs in Windows.
I think your problem may be a little easier to solve. You just need to plug them all in at once and use something where you can specify which device to use instead of tying each to a particular output method's default. In linux with mplayer you can easily do that with ALSA just by using -vo alsa:device=hw=1.0 (changing 1.0 to choose which main device and subdevice to use.) I'm not sure how this could be done in Windows though as it seems dsound and win32 won't take this parameter (or more accurately the player is likely just plain not compiled in the versions I've downloaded with any such option. Maybe a modified version out there will do that.) Anyway, even if mplayer can't do it there must be some console player that can do it out of the numerous ones out there. Just if there is any that supports the multitude of Windows output options I'm not really aware of it (I just see waveout and dsound out or most things.)
 
The easiest method would be to play the same sound and record it via line in, or stereo mix to exclude hardware output/input stages, and then compare the results. RMAA would come in handy.
 
The input is already exactly the same across everything. It's switching outputs that is the problem. I need to test the actual output methods Windows itself utilizes. ASIO, DirectSound, and two different WASAPI methods just for instance. I'm just trying to eliminate variables. The problem is, I need a way to actually compare the outputs in a way that eliminates all possibility of bias like the way ABX tests eliminate bias in testing multiple inputs.

EDIT: I completely misread what you said. Your idea has merit. I'll have to think on how I might do that with what I have now. Though if I confirm that there are differences, it doesn't confirm one way is "better" than another just from a simple comparison. Perhaps if I play each via the RPi. I'll have to see if there is an ABX compare tool for Linux. I think it would have to be the line output rather than stereo mix since that likely grabs the sound too early in the process (and surely can't even get things like ASIO at all.)

Thanks for the idea.
 
You could play a song and compare the original file vs recorded waveforms and see which one matches the best. Audacity probably has some tools for a quick comparison.

Foobar ABX-ing, after normalizing the files and cutting them to same start/end, would also be a good method.
 
You could play a song and compare the original file vs recorded waveforms and see which one matches the best. Audacity probably has some tools for a quick comparison.
A nice idea, but I think this will be impossible. The differences will be, overall, too subtle and all recording methods that would work (stereo mix is just not going to do it) will be subjected to picking up random noise along the way as well. I don't have exactly the best recording equipment here.

Foobar ABX-ing, after normalizing the files and cutting them to same start/end, would also be a good method.
Presumably with whatever is going on it could disguise the issue in the same way. I guess that would mean whatever effect it has would be doubled so presumably more noticeable but this is a highly subjective thing as it is, so I harbor doubts it could be this easy.

I guess the RPi is the closest thing I have to a control here. It wouldn't be a blindfold test on its own though. A simple solution if I could figure out a way to do it is to come up with some sort of method that randomizes the filenames where I can somehow derive the originals when needed (eg if it output what it actually did to a text file or something.) Easy enough to do -- maybe not even requiring an actual script -- but I'm not really sure how to randomize the filenames.

I may just have to brute force all this. The idea of recording them and putting them on something more of a control (eg the RPi I guess) still at least lets me eliminate the process of actually switching output methods (which requires me to stop it, change the setting, apply, then start the file over. This way I just change to a different file and that's it.) Perhaps this might be sufficient since I could switch around significantly faster.
 
Just send them to a friend and let them change the names randomly to whatever.

Whatever you end up doing, post the results. I'm interested if you'll find differences, though I don't believe you should unless something's wrong.
 
Back
Top