Sound in storytelling
To start of ’a new beginning’ with a blog post about what I proclaim to be in the process of leaving behind, might not make a lot of sense. But in a way, it does. At the moment, video doubtlessly is one of the fields that I know the most about, technically and intrinsically. And such, applying cognition on it, might not be such a bad idea. It was in any case that kind of thinking that brought me to where I am today.
Not too long ago I saw this video made by a Simon Bradley on Vimeo, that he posted in a cinematography group. Consisting of beautiful footage of Iceland. See for yourself…
Iceland – The Land of Legends from Simon Bradley on Vimeo.
And while beautiful, what caught my attention wasn’t really the footage itself. But the sound. Or rather, the concept of ‘supernatural sounds to create immersion’ (if the term has not yet been coined, I’m doing it now). The idea is not new. Supernatural sounds are part of basically every movie that you’ve ever seen. No matter if you are aware of it or not. The process of creating them is called foley.
For example, you could have a look here to see how certain sounds are created in movies:
If you didn’t already know, now you do.
Either way. Usually, these sounds. The supernatural sounds, are in 99 our of 100 cases created to highlight human interaction, or human interaction with the environment. Or a car accident. Or a tsunami. Or just whatever that is on screen and/or in center of attention. However, what struck me with this Iceland movie was the supernatural sounds of the environment. The first “baaah”, from the sheep was my cue, my attention was caught. Why? It felt as it didn’t belong. Which could be a simple mistake, as there’s for example the sound slightly resembling a plane a few seconds before that scene, which I didn’t react to on my first watch. The point is; the sheep sound is a sample, either from own material or somewhere else. But not that given moment. If you listen carefully it sounds like ‘more’ than reality.
And it highlights something, that I think is of particular interest. What do the environment sound like, and do the environment always need to sound natural? Because, while not easy to try to re-imagine the sound of silence for a certain scene (let alone actually creating it), it’s clear to me, that sound, apart from few and far between highlights, is extremely underused as a storytelling tool. Sure, content creators have started to understand the importance of sound. But usually that just means the importance of having “good enough” sound, not getting in the way of the visual story.
Time for a concrete example? Have a look at 01:36 in Simon Bradley’s Iceland-video. I’m quite certain that the ice do not sound like that in that particular shot. The waterdrop around 2:37 is another example. And then the waves at 03:10; while they may actually sound like that, they feel more present than what I would’ve expected from a truly natural sound mix. There’s something particularly satisfying to it. Don’t you think?
This sprung the thought in me, which might be a bit crazy, but either way. What would the cameras interaction with the environment sound like? Or the entity that is the camera. If we existed in it, or as part of it. What would we hear?
I don’t know. But just the sheer thought of keeping that as a mantra through a production sounds very intriguing to me.
There is another, somewhat older video that Simon may have taken inspiration from.
Watchtower of Turkey, by Leandro Dalessandri:
Watchtower of Turkey from Leonardo Dalessandri on Vimeo.
Which to me, still stands as something of a ‘state of the art’ piece when it comes to video editing for video editings sake. It realizes many techniques at a more than good enough level and concatenates down to a fantastic “travel video”.
Sound plays a big role here as well. And while the sound design may consist of a little too many “swooshes” it still coheres very well with the imagery. Which is mostly consisting of visual ‘swooshes’, in editing but also in terms of movement in portrayed subjects. Around 0:48, while in reality it obviously does not make a sound, we still hear the chandelier, and I get struck by the same satisfactory feeling as before.
One could look at this as one simple run of the mill technique that when adopted by many, will turn into a cliché. And while that is probably true, that is not my point. Rather, I want to raise the thought, not only to ‘add’ good sound to production for increased production quality. But to use sound as ‘part’ of an experience. Creating immersion by sound, using techniques (like this one), that is not exclusive to the very high end of the spectrum.
Because in this day of age, with the tools to have immersive experiences, available – right now. There is a clear gap between the capabilities and what is actually produced for the platforms. Yes. I am talking about VR. But also what is to come. And I think that no matter which screen we are producing for, there is a lot more potential we can realize even with current technology, that we yet have not.
And of course, in most cases it comes down to production costs. Which is the ugly truth, the pace is so high in most TV/Commercial/Cinema productions, that there’s no time use any out of the box techniques except for productions made with pure love for the art or possibly the very high end of commercial productions.
But production costs is also due to the currently available tools. And I think we live in exciting times at the moment. The development however is not necessarily driven by the traditional film industry, but rather the game industry, which are trying to bring more and more accessible tools for sound synthesis as we speak.
About half a year ago I went to a seminar with J.P. Lewis that talked about his work at EA for the moment, which entailed their work on speech synthesis. He didn’t say much about how far they had gotten with this technology, but my guess is that they will roll it out successively. Trying it out on less important game characters in the beginning – which will most likely save lots of time and money.
And as time and money really is key here. Technology like this will most likely result in increasing production quality while cutting costs in departments that’s not essential to gameplay (voice actors, foley). And while this is very much proprietary technology right now, there’s most likely other companies working on similar things.
As a sidenote, if I’ve captured your interest with all this, I also found this course, which at least I’m definitely intrigued to take on:
https://online.stanford.edu/courses/sohs-music0002-physics-based-sound-synthesis-games-and-interactive-systems
