How do we analyse sound?

In order to analyse the relationship between musical sound and body motion, it’s important to have some way of analysing and describing musical sound. In this video, we will briefly introduce some useful tools for this purpose. When talking about music in everyday life, we use everyday words to describe the musical sound– groovy, heavy, happy, noisy, funky, romantic, and so forth. Notice that many of the descriptions we use are somehow related to highly subjective, non-musical aspects. For instance, someone may label a certain piece of music romantic, while others might not associate romance with the same piece. Other descriptions are slightly less subjective, although still not a precise description of the musical sound.

For example, genre related descriptions like funky, metaphoric descriptions like punchy, or descriptions of how we perceive the sound such as loud or soft.

The descriptions of sound mentioned so far are verbal descriptions. They are words or labels that we use to describe a sound or an entire piece of music. Sometimes, such descriptions are what researchers are interested in, since verbal descriptions reveal something about how people experience the sound, like their associations, et cetera. However, there’s often a need for more objective descriptions of the sound signal itself.

In order to study the sound signal more closely, we shall first have a look at three different ways of representing the sound signal. These are the waveform, the spectrum, and the spectrogram, each of which gives us some visual indication of what goes on in the sound signal. Later, we shall have a look at how we can extract more information from these representations. The figures that I’m about to show were made in Sonic Visualizer, which is a user-friendly and free piece of software that I would recommend you have a closer look at on your own.

This figure shows a waveform representation of a sound signal, with time on the horizontal axis from 0 to 7 seconds and amplitude on the vertical axis. From this representation, it is easy to spot the duration of the sound and the periods of silence. It’s also possible to get an idea of the dynamics of the sound, although precise or detailed information is difficult to read by inspecting the waveform. Let’s listen to the sound

This figure shows the waveform on top and the spectrum of a short part of the sound on the bottom. With frequency on the horizontal axis and amplitude on the vertical axis, we get information on the frequency content of the sound. For a single tone, the fundamental frequency and the range of overtones are easy to identify.

A spectrogram representation shows how the frequency content evolves over time. The horizontal axis shows time and the vertical axis shows frequency, from the bass on the bottom to the treble at the top. Amplitude is shown by colour. The figure on the right shows the same saxophone melody, and how the fundamental frequency and overtones change over time. Let’s listen to it.

Now we have three representations of the sound signal. Already from these representations, we can start to say something about the sound. The amplitude of the sound wave and the frequency content are both varying over time. Still, the representations we have looked at contain large amounts of information, and sometimes it might be necessary to narrow down our description even further. Sound is generally too complex to describe fully with a single number or graph, but a good description can be given if you decide to just look at a single aspect of the sound at a time. Such an aspect is what we usually call a sound feature, or sound descriptor.

A descriptor is a numerical description of some quantifiable aspect of the sound. Many existing features are quite intuitive, as they are either based on simple transformation, so the waveform or the spectrum, or perceptual models. Other descriptors are more technical, but have proven useful in certain types of analysis. Descriptors may either be global, meaning that they describe an entire sound, or they are time varying, meaning that they are calculated from short, sequential, time frames, and then forming a sequence of numbers. Using tools like the MIR Toolbox from MATLAB, it’s possible to extract a range of descriptors from a sound file. Here, for instance, we see how the sound energy of the saxophone melody changes over time.

Other examples of some descriptors are tempo, often described in beats per minute; pitch, denoting the fundamental frequency of the sound; spectral flux, describing how much the spectrum changes; roughness, using a mathematical model to describe roughness that occurs because of limitations in our auditory system; spectral centroid, which denotes the barycentre of the spectrum, or in other words, the point of the spectrum where there’s an equal amount of energy above and below. Let’s have a look at an example of how the spectral centroid changes when the brighter parts of the sound are filtered out. This example shows an analysis of a drumbeat, with the waveform on top, spectrogram in the middle, and the spectral centroid at the bottom.

So to close up this lecture on sound analysis, let’s ask the question what numerical sound descriptors may be useful for. The obvious reason is that numerical descriptors tell us different things about the sound signal than what we are able to hear ourselves. They may be used to compare large sets of sound files, and as such, organise sound databases. Furthermore, numerical sound and movement descriptors may provide us with partial answers to big questions like, how do people associate body movement with sound? Is there a correlation between sound energy and movement energy in dancers? Or between musical tempo and movement tempo?

Or what about pitch, which we describe as going up or down, how precise do we actually relate it to physical vertical movement? Armed with numerical descriptors, it’s possible to do systematic studies with statistical analysis and answer such questions.