Skip main navigation

Visualising video

How can you see through time? This article presents video-based visualisation techniques developed for the analysis of music-related body motion.
© Alexander Refsum Jensenius, University of Oslo
Here in Music Moves we present a number of different types of motion capture technologies. Many of these are excellent and work well for their purposes. Still, however, video recording is most likely the most accessible “motion capture” technology for most people. Video cameras are nowadays easily available everywhere, so anyone can get started right away.
It may be odd to think that it is necessary to create visualisations of a video recording. After all, video is visual to start with. However, watching a running video is not a very efficient way of analysing large sets of video recordings.

Motion images

One of the most common techniques when one works with motion analysis from video files is to start by creating what we call a motion image. The motion image is found by calculating the absolute pixel difference between subsequent frames in a video file, as illustrated in the figure below. The end result is an image in which only the pixels that have changed between the frames are displayed.
Motion image
The quality of the raw motion image depends on the quality of the original video stream. Small changes in lighting, camera motion, compression artefacts, and so on can influence the final image. Such visual interference can be eliminated using a simple low-pass filter to remove pixels below a certain threshold, or a more advanced “noise reduction” filter, as illustrated below. Either tool cleans up the image, leaving only the most salient parts of the activity in the motion.
Filtered motion image
The video of the filtered motion image is usually the starting point for further processing and analysis of the video material.

Motion-history images

4.7

28 Reviews
A motion image represents the motion that takes place between two frames but does not represent a motion sequence that takes place over more frames. To visualise the motion itself over time, then, it is necessary to create a motion-history image—a display that keeps track of the history of what has happened over the course of some number of recent frames. One approach is to simply average over the frames of an entire recording. This produces what could be called an average image or a motion-average image, such as shown below.
Average image
These images may or may not be interesting to look at, depending on the duration of the recording and the content of the motion. The examples above are made from a short recording that includes only one short passage and a raising of the right hand. The lift is very clearly represented in the motion-average image, whereas the average image mainly indicates that the main part of the body itself stayed more or less in the same place throughout the recording.
For longer recordings, in which there is more activity in larger parts of the image, the average images tend to be more “blurred”—in itself an indication of how the motion is distributed in space.
To clarify the motion-history image, it is possible to combine the average image and the motion-average image, or possibly incorporate one frame (for example, the last frame) into the motion-average image. The latter alternative makes it possible to combine a clear image of the person in the frame with traces of the motion-history, as illustrated below.
motion-history image
Motion history images may be usefl to study, for example, performance techhnique. The figure below shows a visualisation of a percussion study. Here, each image represents an individual stroke on the drum pad, and the image series serves as a compact and efficient visualisation of a total of fourteen different strokes by the percussionist.
Percussion
Each of the displays in the figure above represents around fifteen seconds of video material. As such, this figure is a very compact representation of a full recording session.

Motiongrams

The motion-history images above reveal information about the spatial aspects of a motion sequence, but there is no information about the temporal unfolding of the motion. Then a motiongram may be useful, since it displays motion over time. A motiongram is created by averaging over a motion image, as illustrated in the figure below.
Schematic motiongram
This figure shows a schematic overview of the creation of motiongrams, based on a short recording of a piano performance. The horizontal motiongram clearly reveals the lifting of the hands, as well as some swaying in the upper part of the body. The vertical motiongram reveals the motion of the hands along the keyboard, here seen from the front, as in the previous figures.
One example of the ways in which motiongrams can be used to study dance performance can be seen below. This display shows motion-average images and motiongrams of forty seconds of dance improvisation by three different dancers who are moving to the same musical material (approx. forty seconds). A spectrogram of the musical sound is displayed below the motiongrams.
Dancers
The motiongrams reveal spatiotemporal information that is not possible to convey using keyframe images, and they facilitate the researcher’s ability to follow the trajectories of the hands and heads of the dancers throughout the sequences.
For example, the first dancer used quite similar motions for the three repeated excerpts in the sequence: a large, slow upwards motion in the arms, followed by a bounce. The third dancer, on the other hand, had more varied motions and covered the whole vertical plane with the arms. Such structural differences and similarities can be identified in the motiongrams, and then studied in more detail in the original video files.

From Music Research to Clinical Practice

We can make a little detour at the end of this article. As researchers working on basic issues, we are often asked about the “usefulnes” of what we do. It is often difficult to answer this question, because our research is not meant to be useful in the first place. But sometimes seemingly “useless” developments can have an impact elsewhere.
The visualisation techniques mentioned above have actually turned out to be very useful in medical research and clinical practice. A group of researchers in Trondheim, Norway, found that the motiongram technique was an excellent way of detecting so-called fidgety motion in infants. This is important when it comes to screening pre-term infants that are in the risk zone for developing cerebral palsy, as shown in this image with a healthy infant (top) and an infant with cerebral palsy (below).
Infants

Tools

  • VideoAnalysis. A simple application that lets you import av video file, press a button, and you get motion images, motiongrams, and more.
  • Musical Gestures Toolbox. This toolbox is available for the programming environments Max, Matlab and Jupyter, and provides the building blocks for making many different types of video visualizations.

References

© Alexander Refsum Jensenius, University of Oslo
This article is from the free online

Music Moves: Why Does Music Make You Move?

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education

close