By now the majority of us will have heard some combination of these letters, the most likely one being VR and usually accompanied by the phrase "Oh yes! The goggles!".
For those who don’t know, here’s a brief primer on what the differences are:
VR - Virtual Reality: The one with the goggles. A headset is worn by a user in order to take part in an experience. You are taken completely out of reality and put in a virtual world.
AR - Augmented Reality: Visuals are put on top of the real-world, augmenting the real world around the user. There are headsets that do this, the HoloLens for example, but most commonly it is used on mobile phones (Snapchat face filters, Pokémon GO – that is AR or very similar).
XR - Extended Reality: A term used to combine immersive experiences that use VR, AR and other human-machine interactions from technology and wearables.
MR - Mixed Reality: An experience that combines both elements of VR and AR where real-world and digital objects interact.
But those super-stylish goggles are only part of it - sound plays a huge part too.
For VR experiences, the idea is that a user should become fully immersed in a virtual world. Completely removed from reality. In this type of experience visuals are heavily relied upon by the user to make them believe that they are in a different world. If the virtual world is not reacting seamlessly to the users actions then they will not be fully immersed. Similarly the main challenge of the sound design is then to ensure that the sound marries the environment completely. Much like traditional linear films, the sound-design and Foley must match the visuals otherwise you end up with a poor product (or very amusing if it ends up being so bad it is good) but, for the most part, that is not the desired outcome. And again, like traditional linear formats, bad sound design or 'wrong' sounds, stick out like a sore thumb whereas good sound design should go unnoticed.
The biggest difference when sound-designing for VR is that because it is a virtual world, there is rarely anything linear about it so you can not just do a track lay, run it off and play it. You have to design for specific interactions or moments. Or for longer pieces of audio, for example atmosphere, there is not always a predetermined amount of time that you will be in one scene. So you have to design it to be able to continue for long periods of time, without getting repetitive or dull, but at the same time it has to be able to be stopped at any point. Or have extra layers brought in at any point. And still make sense in the environment.
The biggest challenge when sound designing for VR experiences is real-time user interactions. Again because nothing is linear, you can not just design a long bit of audio with perfect fades and automation. That all has to be calculated and done in real-time in the experience. For example say in an experience a user had to roll a ball that had bells in it.
You do not know how hard the user will throw the ball so you do not know how long the ball will roll or how fast it is going. The challenge as a sound designer is to create a sound, or a bed of sounds, that can last less than 1 second (for the clumsy who may accidentally drop the ball, like myself) but can also last a long time. And at the same time it has to be designed real-time in the experience to react to the physics and movements of the ball, e.g. pitch and volume reacting to how fast the ball is moving.
When it comes to AR experiences, as the name suggests, it is about augmenting the world around you rather than completely immersing the user in a new environment. Augmented reality is still a relatively new technology so pure AR experiences are only starting to emerge or be explored. There are AR headsets which come with built in audio features such as the HoloLens or the new Magic Leap, but at the moment most AR content is viewed via mobile or tablet just through the devices speaker. With VR you can usually rely on the user to be wearing headphones which allows for the sound design to shine and take over, immersing the user in the new environment.
So, as well as the challenges of non-linear content like VR, the other challenge when designing for mobile AR experiences is you have to bear in mind the users physical environment.
The sound design from the phone has to complement the users physical space rather than replace it, whilst at the same time being detailed enough to convince the user that there is something else in their environment despite the fact the user can see the real world.
A fully immersive experience requires all senses to be 'fooled' into thinking they are somewhere they are not. In VR and AR, visuals play a massive role as it is what a user focuses on. The virtual world needs to work and respond in a way that makes them believe that what they are seeing is real. However, as the way we hear and experience sound just in our day-to-day lives is so ingrained in us as humans, we are subconsciously aware of what we are hearing 24/7. When a sound feels out of place we notice it immediately. Therefore if the sound does not match or respond to the virtual environment, no matter how pretty the virtual trees or buildings are, the user will not believe they are in a virtual world and the immersion would break. Which is why sound is such a core part of immersive experiences and needs to be crafted properly in both VR and AR.
Written by Mirren Malcolm-Neale, GCVRS Sound Engineer
Instagram - @grandcentralrecordingstudios
Website - www.gcvrs.net