In honor of the much-anticipated release of a particular sequel next week, I decided to center my discussion of video game audio around one of my favorite video games in recent history: Mass Effect 2. Composed by a sound team led by the illustrious Jack Wall (Unreal 2, Myst IV, Jade Empire, etc.), the soundtrack is a respectable collection of music on its own, its diverse and energetic music setting it apart in a world of space operas. However, in-game, amid the ever-evolving action and drama, is where this soundtrack shines most. The Mass Effect 2 soundtrack enhances the gaming experience in many ways, but most interesting to me is its stellar application of one of the most popular and exciting topics in current video game audio research: the concept of dynamic music."Dynamic music," generally speaking, refers to music that changes in real-time due to interaction with the audience. Thus far, dynamic music is mostly confined to the world of video games; unlike other forms of media, such as film and radio, where there is very little or no interaction with the audience, the "audience participation" element inherent to video games makes this medium a perfect vessel for this form of interactive audio. Speaking from an engineer's point of view, this so-called "nonlinear" audio can be modeled as a sort of feedback loop (see below diagram). Whereas most traditional forms of audio in media have an uninterrupted stream from the transmitter (the musical composition) to the receiver (the audience), interactive audio introduces an additional projection from the receiver back to the transmitter, granting the audio the ability to evolve in real-time based on the audience's behavior.
While dynamic music in video games might sound like a recent innovation, it has in fact been present for much of video game history. The PSG-, FM- and MIDI-based sound cards of older consoles all generated tones in real-time, like an electric keyboard. While difficult to compose for, these systems made it very simple to manipulate the music dynamically. One might even argue that dynamic music was introduced in the very first game with a true soundtrack, 1978's Space Invaders, whose famous descending tetrachord speeds up as more enemies are eliminated. A clearer example is the 1982 game Dig Dug, whose musical content progresses only when the player is moving. Perhaps a more recognizable example can be seen in the Genesis-era Sonic the Hedgehog games; whenever the player collects a set of speed shoes, the tempo of the music would increase for a certain amount of time. In fact, even if the background music itself changed during this time interval, the tempo nonetheless remained elevated until the speed shoes' tenure expired.
An important turning point in video game music history came in the mid-1990s with the switch to Redbook audio, better known as CD audio. In all conventional respects, Redbook was a big step forward for video game music; no longer were video game composers restricted by the sounds that the consoles' sound cards could generate, and this led to the capability for live orchestral soundtracks such as that of 1997's Total Annihilation. However, Redbook audio was space-intensive, and because it was pre-recorded, it could not be manipulated in real-time in the same manner as the older audio technology. Thus, while compositional styles no doubt flourished, the dynamic music found in many previous-generation games effectively ground to a halt.In more recent generations, however, increasing computing power has made it possible for dynamic music to reemerge. Certainly, the obvious examples are video games in which interactive music is the focus, including such games as Electroplankton and the Guitar Hero series. Even for non-audio-focused games, however, sound teams have devised clever ways in which to introduce dynamic elements, ranging from simple techniques (fading, "stinger" chords/passages) to more compositionally challenging ones (cue-to-cue transitions, layering). While newer approaches to dynamic music have yet to rival the level of interactivity in the 8- and 16-bit generations, it is certainly headed in a promising direction.
This leads us back to Mass Effect 2, whose soundtrack contains one of the most effective, yet simplest, implementations of dynamic music I have encountered. The dynamic aspects of Mass Effect 2's music can be most readily appreciated in the battle missions, constantly-changing environments where the amount of action at any time is largely controlled at the player's own pace. Because of the wide range of activity in each of these levels, from quiet strolls to explosion-filled firefights, it seems essential to have music that can reflect these frequent mood shifts.Mass Effect 2 accomplishes this dynamic behavior by adjusting the music to reflect the amount of onscreen activity. (Some speculation follows, I admit, but I think it's a correct simplification of what's going on.) At any point in time, the game examines the total amount of activity currently present and assigns that value to one of several discrete "activity levels." In turn, each "activity level" corresponds to a particular musical texture, which is switched in real-time as necessary.
At the most fundamental level, this technique is equivalent to layering--increasing activity levels correspond to an increase in textural complexity. However, rather than limit themselves to simply adding and removing voices, the Mass Effect 2 composers wrote individual pieces of music corresponding to each activity level. Below are some examples from one particular battle, arranged in increasing activity level:
If not composed carefully, using a separate piece of music for each activity level can lead to some awkward transitions. However, the cohesive compositional style of the entire set prevents this from happening. You may have noticed that the bass pattern stays constant in all four tracks, providing a powerful foundation upon which the remaining texture is built. The same is true of the harmony (or implied harmony, as the case may be). The other voices may change substantially between activity levels, but by holding strong stylistic and structural elements constant, the composers achieve a remarkable sense of consistency. No wonder the in-game transitions sound so flawless.
The best way to appreciate any concept is to see it in action, so here's a clip of me playing one of the combat missions in Mass Effect 2. I've labeled each of the activity level changes encountered in this clip; try to listen for the musical changes during each transition. Also, please forgive my terrible playing; I admit that my gaming skills haven't been the best of late.
For those interested in exploring these activity levels some more, I've created an application where you can manually switch between the four activity levels and listen to how the music adjusts dynamically. Unfortunately, I couldn't get it working as an Internet applet, but below is a download link if you're interested in poking around.
I think we stand at a very exciting point in the story of interactive audio. Modern gaming technology such as the Kinect offer novel methods of feedback to interact with game audio, and a number of existing compositional techniques, such as granular synthesis, hold exciting possibilities for dynamic music. For the moment, however, dynamic music is still emerging, and I'll be pleased to see any game make a valiant effort to introduce some dynamic elements in its soundtrack. If the result is as effective and eloquent as in Mass Effect 2, all the better.
Questions? Critiques? Something I didn't get right? Something you'd like me to elaborate on? Please drop a comment below!
For more information on video game music research and interactive music, please refer to the archive of video game music publications at GamesSound.com, particularly this article on nonlinear video game audio.
Happy week-before-Mass Effect 3! :)
xds
