For Oerst’s full biography, and insight into what his job involves, see:
Lennart Nacke
In this part of the interview, we talked about the design and elicitation of specific emotions in viewers. From a researcher’s point of view, this is fascinating – in games research, we often want to see how certain aspects of games affect player experience, including their emotional responses. Personal experience has shown me that some emotions are easier to cause than others (unfortunately, frustration is easy to elicit accidentally). For these reasons, I was particularly interested in what Orest has to say about designing for specific emotions through his work with audio.
KR: How do you go about making a specific atmosphere through audio design? Where do you start? Is there a guideline for inducing specific emotions?
OS: Atmospheric design (be it background ambiences or specific sound effects) can be derived organically (through recording various elements in the real world) or can be generated artificially (digitally) through signal generation and processing. There are countless tools for the latter. It really depends on the feeling you’re trying to create in the scene. A range of emotions can be affected with either approach.
As mentioned, a lot of games research on audio (of which there still is a scarcity in general), has focused on horror games. This is mainly because the emotions targeted and elicited by horror games are fairly strong ones – anxiety, suspense, fear. Audio plays a big part in inducing and upholding these emotions, and they are also easier to measure than many others. With his experience in mixing sound for a wide variety of genres, from horror movies to documentaries, Orest has valuable insight in comparing audio for different genre conventions:
OS: The mantra in comedy sound has generally fallen across two overarching tenets – the timing of the sound, and the loudness of it with respect to picture. Both attributes contribute to the overall “funniness” of the film. With horror films, the same can apply, although I personally feel the most effective experiences are delivered to a viewer when not every sound plays loud all the time. Again, this is where I think David Cronenberg is such a master in storytelling. Another director I’ve worked with is who is also a very astute storyteller is Robert Eggers who crafted a brilliantly unique and very disturbing horror film entitled “The Witch”. With both directors, underplaying certain elements in a mix (in terms of sonic shape and level) can draw the viewer in to the moment and allow them to become vulnerable. The shock and horror come with an about-face in the sound that reinforces or juxtaposes what’s happening visually. Both directors have a unique ability to make you experience and look at things that you don’t want to look at – but you look anyway. They use sound to help draw you into that place – that experience.
There is also a difference in challenge in designing the audio, depending on the tone, volume and target emotions:
OS: In a way, from strictly a sound standpoint, I have found the quiet, more reserved dialogue-centered films to be the most challenging to mix. In my own experiences, I have found more often than not that creating loudness in soundtracks requires a different sort of articulation in one’s work than making that same environment quieter in a scene. If a director wants to hear only their original production dialogue track without the artifice of too many added sound effects or any musical score, the presentation of those few elements remaining need to come across with immaculate detail in terms of the aesthetic, especially if the quality of the production quality dialogue is compromised. That’s not to say that louder complex passages don’t require the same detail. Rather, the fewer elements you have, the more concise the fit has to be. There is nothing sound-wise to hide behind or mask potentially noisy production dialogue tracks in the few key elements you may be presenting, when dialogue is the main driver of the scene (versus an action-oriented scene).
In our skype talk, Orest talked about conveying the directors’ visions (see also the previous part of this interview series). Given that sound design is inherently audio-based, I was particularly interested in how Orest would describe communicating these visions for audio design with directors, and how they would pinpoint why a certain sound design works or doesn’t work:
OS: Emotions are often a principle component in the conversation with directors. How do we want the audience to feel? In much the same manner a director has a vision for the look and aesthetic feel of a film, they often have a vision for the soundscape as well. This can be more challenging to articulate specifically as sound serves as the invisible “wrapping” around the visual component. Being 50% of the presentation, sound is equally powerful (and often times more powerful) in conveying a certain message or intent to the viewer, so emotions can describe how we want the audience to feel. Do we want them to feel scared, shocked, sad, happy, etc.? […] Prudent sound choices can help direct our emotions very succinctly one-way or the other. Beyond choice of sounds, the timing of hearing certain sounds can be equally effective in delivering intent. In some cases the ear may lead the eye – or the eye may lead the ear.
In games, this may in some cases even go a step further, with the ear leading the player’s behaviour. In the studies I conducted at Waterloo last year, we explored the effects of audio in a commercial horror adventure game, comparing a VR version and a PC version of the same game (Rogers et al. 2018). While we found that the effects of audio were diminished in VR (see our paper for more discussion on this), we also found that people’s reactions were highly subjective. For some players, the audio was so influential as to change their behaviour in the game: when they heard “creepy” music, they felt anxious and tense, and decided to choose a different path through the game world. While this clearly does not work for all players, and seems to be overshadowed in VR by novelty bias and the sensory experience, this still shows that audio design is an important factor in game development.
In the next and final part of this blog series, we will look into the differences between movies and games in terms of audio design.
Written by Katja Rogers (Ulm University), visiting researcher at the HCI Games Group.
Dr. Katja Rogers is a graduate of the Computer Science Ph.D. program of the Institute of Media Informatics at Ulm University, Germany. In her graduate studies, she was a visiting scholar with the St Andrews Computer-Human Interaction (SACHI) research group in Scotland, spent a semester abroad at the National Taiwan University, and undertook two research visits with the HCI Games Group in Waterloo. As a postdoctoral researcher within Dr. Lennart Nacke's research group, her research interests and current projects at the HCI Games Group focus on the effects of audio on player experience, embodied interaction in mixed reality, how games and digital technologies can improve wellbeing and fidelity/realism in virtual reality.