One of many reasons fans of the Daredevil comic book were bummed by Mark Steven Johnson’s 2003 silver screen interpretation of the character (starring the man who will be Batman, Ben Affleck) was the way in which the Fox flick changed the character’s chief superpower, his radar sense. Daredevil is blind, but he gives off a radar that allows him to sense outlines of objects and people all around him. He may not know what color the baseball bat is, but he can sense it coming from directly behind his head. The film changed that to exceptionally precise hearing, which isn’t as cool as giving off an organic radar. That said, it looks like the film’s interpretation reflected one of those superpowers observed here in the real world.
WNYC’s Invisibilia podcast interviewed a blind man who can use his hearing to see in much the same way as Mr. Affleck’s superhero does, maintaining a very active lifestyle, riding bikes, running races and hiking mountains, without any sight at all. Not only that, he trains other visually impaired people to tap into this latent ability as well. Even if sighted people don’t have nearly such refined senses of hearing, we intuit more information about space from what we hear than we realize. That’s why getting the direction of sound right is so critical to immersive virtual reality.
The Observer spoke with Jacob Adelgren, the audio specialist at the Netflix-for-VR startup, EEVO, who spends all day every day working on how the company’s VR composition system can do a better job delivering not just high quality sound, but the little frequencies that tell our brains which direction the sound is coming from, as our virtual bodies move through digital spaces. (Speaking of superpowers, the company recently backed a recent VR film that gives a virtual straphanger mind reading abilities.)
“Humans have an ability to localize sound within about a degree,” Mr. Adelgren explained during an interview. There’s a variety of ways in which our brains interpret location information with our ears. For example, “There’s a slight sound delay between your two ears,” he said. If a sound source is distinctly on one side of a person, he won’t consciously perceive the slight difference between the moment it arrives at each ear, but his brain will. It gives part of that sense of location.
That’s why people without sight were so excited about this two-microphone headphone recording system, saying it was the first home movie camera for the blind.
That delay isn’t the only clue we pick up from sound. “Different frequencies attenuate differently around our heads,” he explained, so what parts of the full sound do or don’t come through also help our brains locate it. EEVO is working with a couple of different open source libraries that have measured how sounds travel around human heads, using very sensitive, very precise microphones built into mannequins. The sets of “head-related transfer function” measurements he’s working with are the CIPIC and another from the MIT Media Lab.
With those measurements, he can play sound for VR viewers in the subtly different way they would sound, based on the way that the person’s head is oriented in the movie or experience, which helps the person to unconsciously locate it, making it feel more real.
“I think filmmakers do need to understand that the audio in a VR film is integral to how immersive it is,” he said. To make it work best, he needs filmmakers to do a little more work during their shoots. Audio systems like EEVO’s work best when they can work from impulse response measurements in the actual room where a scene was filmed. The measurements are simple to make, though. The best ones make what he called a “sine sweep,” a simple sound that covers the full audible spectrum in a moment. The device measures what bounces back in a given location. If a filmmaker takes a sine sweep in a few spots, the system can interpolate in between them. Mr. Adelgren can also model a space, but it won’t be as good. “There’s a world of difference between taking a measurement in a real space versus a virtual space.”
In a pinch, though, there’s an app that can take decent measurements on an iPhone, he said.
Right now, Mr. Adelgren and professionals like him are working from models that assume a sort of standard human (or standard latent superhuman), but the shape of our heads, the size of our ears, the width of our ear canals and that weird bump thing on the front of our ears (it’s called the tragus) all play subtle roles in shaping the sounds our brains interpret. So, looking forward, he said, “Eventually, my goal would be to find out more info about the person and we could create a more customized filtering scheme.”
With that, EEVO can optimize the virtual reality experience to the unique Daredevil inside us all.
Thoughts on this story? Email firstname.lastname@example.org or on Twitter at @BradyDale