Binaural / Holophonic / 3d virtual sound

S

serpretetsky

2[H]4U
Joined
Dec 24, 2008
Messages
2,180
What do you guys think about the idea of implementing binaural recording audio engines into games?

examples (you must listen with headphones, speakers wont work):
http://www.youtube.com/watch?v=x5G3HUiscW4&feature=related
http://www.youtube.com/watch?v=8IXm6SuUigI

I know they already do something similar with things like "headphone" option in games, "virtual surround" and cmss-3d. But it's my understanding that these technologies actually take the 5.1 output from games and then create the virtual surround from that (being limited directly by 5.1 output).

It would be cool if there existed an audio engine that the game developers could just feed the audio clip to and the directional position (xyz coordinates, instead of channels) and then the audio engine could correctly apply the algorithms necessary to make the sound come from any angle, and from any distance, recreating those youtube clips. Or do you guys think that would require to much cpu power?
 
I thought in most games you have to select 5.1, and then enable cmss-3d. Doesn't that only give the amount of information that 5.1 speakers are able to convey to the audio engine, which doesn't included distance or up and down?

Im actually a little ignorant on how audio is passed from the game to the sound card. Do games send individual premixed sounds to the audio api, telling the audio api how loud each channel should play the sound? Do they send already mixed audio for each individual channel?
 
serpretetsky said:
I thought in most games you have to select 5.1, and then enable cmss-3d. Doesn't that only give the amount of information that 5.1 speakers are able to convey to the audio engine, which doesn't included distance or up and down?

Im actually a little ignorant on how audio is passed from the game to the sound card. Do games send individual premixed sounds to the audio api, telling the audio api how loud each channel should play the sound? Do they send already mixed audio for each individual channel?
Click to expand...

AFAIK, I think the game's audio engine handles sound in a full arbitrary 3D space (at least any game with a good audio engine). So the game engine knows everything about exact direction/distance, as well as any occluding objects, reverb, etc.

Then the final output stage of the audio engine takes that 3D audio data, and "mixes" it down to what you tell Windows you own. So if you have configured to 5.1 in windows, the audio engine (again, if well-written) is aware of that, and eventually gives you 6x analog signals. Maybe it's the windows audio driver that does this, but I figure something has to be doing it.

On my Xonar U3 sound card, which has built in Dolby Headphone, you tell windows you have a 7.1 speaker system, and so presumably the DH driver is taking the 7.1 data and downmixing it to a surround stereo signal. So I guess it's different from binaural in that it is taking a 7.1 signal and mixing it down, rather than an arbitrarily 3D audio data. Maybe with other implementations such as CMSS3D (which I believe you configure your windows settings to be stereo or headphones) it might be taking "arbitrary 3D audio data" more directly and convert it into surround stereo signals.

Basically though it is my understanding that thees technologies (DH, CMSS3D) use algorithms that take into account what each ear would hear, and mixes the output sound to appropriately handle this, so in essence it uses binaural sound methodology to achieve the effect.
 
havermeyer said:
Current 3D/5.1 audio doesn't hold a candle to binaural audio, imo.

There's a guy with a start-up trying (I don't know if they're even active anymore, really) to sell(?) "a real-time 3D (binaural) audio SDK": http://www.actionreactionlabs.com/ghost.php

His demo here is one of my favorite examples of binaural audio: http://www.youtube.com/watch?v=yN2KEyEtWWY
And his sporadically updated blog is here: http://blog.digdagga.com/
Click to expand...
yeah, what that guy is trying to do is exactly what im talking about. I dont want headphones to try to simulate 5.1 or 7.1 speakers, its dumb, they should be simulating the actual world.

more interesting read:
http://ixbtlabs.com/articles2/sound-technology/index.html
 
I'd want headphones to simulate 5.1 channels. I find this Dolby Headphone type crap to be very lacking. I cannot position anything with it. Even with this binaural crap, I find that it still doesn't enable me to tell if they are simulating something to be in front of or behind my ears. Just take the virtual haircut thing for example. When there are video cues, it helps a tiny bit, because your mind is telling you where the sound may be coming from, but even in reality, with your eyes closed, you can tell where a sound is coming from. I also don't see it working for altitude whatsoever. All I can tell is left or right, not front and back, or up and down.

I think the answer is to use preserved skulls from dead people, with everything intact, and put the microphones where the ear drums are. Just having two microphones even on the outside of a dummy head, not made of flesh, will never be realistic. The outside shape of the ears provides reflections which provide definite cues in three dimensions.
 
Most video games already incorporate HRTF into the audio output when selecting the headphones output option. No need for Dolby Headphone or CMSS 3D.

http://en.wikipedia.org/wiki/Head-related_transfer_function

DirectSound and now OpenAL have always had HRTF built in as do most in-house sound engines. The functions are not that difficult to implement. If you want HRTF for movies, you can simply use ffdshow for your output as it has a good implementation.
 
I have tried all of the related implementations in ffdshow, no dice. It's absolute crap. The only thing these things do is make an ambiance/echo to the sound.
 
The HRTF function in ffdshow definitely works given a multi-channel input. I use it whenever I watch movies through headphones.

The sound positioning is vastly improved in my Sennheiser HD600s over just a down mixed 2 channel signal.
 
SirMaster said:
Most video games already incorporate HRTF into the audio output when selecting the headphones output option. No need for Dolby Headphone or CMSS 3D.
.
Click to expand...
are there any games you recommend that really showcase the HRTF functions really well?
 
Nothing works for me. I can only tell left and right as direction, and distance. There is no differentiation between sounds behind or in front of me.
 
grimster said:
Nothing works for me. I can only tell left and right as direction, and distance. There is no differentiation between sounds behind or in front of me.
Click to expand...

Did you try the matchbox one i have posted in the original post? I have a harder time with the barbershop one, but the matchbox one for me is very clear when the matchbox is shaking behind and when it's in front.
 
serpretetsky said:
Did you try the matchbox one i have posted in the original post? I have a harder time with the barbershop one, but the matchbox one for me is very clear when the matchbox is shaking behind and when it's in front.
Click to expand...

I've tried a bunch of those pre-made ones. They sound pretty good, but everything sounds like it comes from the rear. Nothing ever moves forward beyond my ears in any of those pre-made things or with Dolby Headphone or HRTF stuff.
 
Imagine how complex a true binaural recording is, especially for a head that moves through a virtual world. Then imagine trying to simulate it in software. I bet it is at least as difficult as simulating visual reality.
 
SirMaster said:
Any game on the Source engine. Choose "Headphones" as the output mode.

Battlefield 3. Enable "enhanced stereo" in the audio settings.
Click to expand...
I'm trying counter strike source, and the effect is very subtle, most of the time i can't tell front from behind, but i suppose that could be two things
1) True HRTF requires a lot of processing, and most games just do a quick and dirty job
2) My cheapy logitech headphones, while being adequate for the matchbox example, do not work in many other HRTF applications. I suppose to truly appreciate the effect you should have inner ear headphones too.
 
For really good positional sound you need headphones that already have a really large sound-stage.
 
serpretetsky said:
I'm trying counter strike source, and the effect is very subtle, most of the time i can't tell front from behind, but i suppose that could be two things
1) True HRTF requires a lot of processing, and most games just do a quick and dirty job
2) My cheapy logitech headphones, while being adequate for the matchbox example, do not work in many other HRTF applications. I suppose to truly appreciate the effect you should have inner ear headphones too.
Click to expand...

I've read somewhere that human hearing uses context clues for front and back positioning. For example, if you were to be blindfolded in a field and have 2 friends each standing in front of and behind you about 30ft and clap one at a time, you would have difficulty telling front from back. But then if you start turning your head a bit while listening, your brain will process the front/back positioning better.

If you listen to the virtual barbershop link, I believe that's the reason the person actually says "I'm going to buzz the back of your head now" and at some point, also tells you when he's at the front.
 
Well I must be too smart for the sound then, because despite that guy saying where he is going to be in the sample, it just nowhere. I do have HD 650s, anyway.
 
You must log in or register to reply here.
Back
Top