Visualising video

Here in Music Moves we present a number of different types of motion capture technologies. Many of these are excellent and work well for their purposes. Still, however, video recording is most likely the most accessible “motion capture” technology for most people. Video cameras are nowadays easily available everywhere, so anyone can get started right away.

It may be odd to think that it is necessary to create visualisations of a video recording. After all, video is visual to start with. However, watching a running video is not a very efficient way of analysing large sets of video recordings.

Motion images

One of the most common techniques when one works with motion analysis from video files is to start by creating what we call a motion image. The motion image is found by calculating the absolute pixel difference between subsequent frames in a video file, as illustrated in the figure below. The end result is an image in which only the pixels that have changed between the frames are displayed.

The quality of the raw motion image depends on the quality of the original video stream. Small changes in lighting, camera motion, compression artefacts, and so on can influence the final image. Such visual interference can be eliminated using a simple low-pass filter to remove pixels below a certain threshold, or a more advanced “noise reduction” filter, as illustrated below. Either tool cleans up the image, leaving only the most salient parts of the activity in the motion.

The video of the filtered motion image is usually the starting point for further processing and analysis of the video material.

Motion-history images

A motion image represents the motion that takes place between two frames but does not represent a motion sequence that takes place over more frames. To visualise the motion itself over time, then, it is necessary to create a motion-history image—a display that keeps track of the history of what has happened over the course of some number of recent frames. One approach is to simply average over the frames of an entire recording. This produces what could be called an average image or a motion-average image, such as shown below.

These images may or may not be interesting to look at, depending on the duration of the recording and the content of the motion. The examples above are made from a short recording that includes only one short passage and a raising of the right hand. The lift is very clearly represented in the motion-average image, whereas the average image mainly indicates that the main part of the body itself stayed more or less in the same place throughout the recording.

For longer recordings, in which there is more activity in larger parts of the image, the average images tend to be more “blurred”—in itself an indication of how the motion is distributed in space.

To clarify the motion-history image, it is possible to combine the average image and the motion-average image, or possibly incorporate one frame (for example, the last frame) into the motion-average image. The latter alternative makes it possible to combine a clear image of the person in the frame with traces of the motion-history, as illustrated below.

Motion history images may be usefl to study, for example, performance techhnique. The figure below shows a visualisation of a percussion study. Here, each image represents an individual stroke on the drum pad, and the image series serves as a compact and efficient visualisation of a total of fourteen different strokes by the percussionist.

Each of the displays in the figure above represents around fifteen seconds of video material. As such, this figure is a very compact representation of a full recording session.

Motiongrams

The motion-history images above reveal information about the spatial aspects of a motion sequence, but there is no information about the temporal unfolding of the motion. Then a motiongram may be useful, since it displays motion over time. A motiongram is created by averaging over a motion image, as illustrated in the figure below.

This figure shows a schematic overview of the creation of motiongrams, based on a short recording of a piano performance. The horizontal motiongram clearly reveals the lifting of the hands, as well as some swaying in the upper part of the body. The vertical motiongram reveals the motion of the hands along the keyboard, here seen from the front, as in the previous figures.

One example of the ways in which motiongrams can be used to study dance performance can be seen below. This display shows motion-average images and motiongrams of forty seconds of dance improvisation by three different dancers who are moving to the same musical material (approx. forty seconds). A spectrogram of the musical sound is displayed below the motiongrams.

The motiongrams reveal spatiotemporal information that is not possible to convey using keyframe images, and they facilitate the researcher’s ability to follow the trajectories of the hands and heads of the dancers throughout the sequences.

For example, the first dancer used quite similar motions for the three repeated excerpts in the sequence: a large, slow upwards motion in the arms, followed by a bounce. The third dancer, on the other hand, had more varied motions and covered the whole vertical plane with the arms. Such structural differences and similarities can be identified in the motiongrams, and then studied in more detail in the original video files.

From Music Research to Clinical Practice

We can make a little detour at the end of this article. As researchers working on basic issues, we are often asked about the “usefulnes” of what we do. It is often difficult to answer this question, because our research is not meant to be useful in the first place. But sometimes seemingly “useless” developments can have an impact elsewhere.

The visualisation techniques mentioned above have actually turned out to be very useful in medical research and clinical practice. A group of researchers in Trondheim, Norway, found that the motiongram technique was an excellent way of detecting so-called fidgety motion in infants. This is important when it comes to screening pre-term infants that are in the risk zone for developing cerebral palsy, as shown in this image with a healthy infant (top) and an infant with cerebral palsy (below).

Tools

• VideoAnalysis. A simple application that lets you import av video file, press a button, and you get motion images, motiongrams, and more.
• Musical Gestures Toolbox. This toolbox is available for the programming environments Max, Matlab and Jupyter, and provides the building blocks for making many different types of video visualizations.

References

• Adde, L., J. L. Helbostad, A. R. Jensenius, G. Taraldsen & R. Støen (2009). Using computer-based video analysis in the study of fidgety movements. Early Human Development 85(9), 541–547.
• Jensenius, A. R. (2007). Action–sound: developing methods and tools to study music-related body movement. Ph.D. thesis, University of Oslo.
• Levin, G. (2005). An informal catalogue of slit-scan video artworks.
• Marey, E.-J. (1884). Analyse cinématique de la marche. cras, t. xcviii, séance du 19 mai 1884.