Video analysis

[ELECTRONIC MUSIC NOTES PLAYING]

Previously, we have looked at different types of advanced motion capture systems, like the optical marker based system and also they inertial systems. Now we’re going to look at how we can use regular video cameras for analysing music related movement. And when I say a regular video camera, that can be anything from the simplest type of webcamera like the one we have here, which is a consumer camera that you can easily attach to your computer. And it’s also the same thing as is typically built into your computer. It can also be a somewhat more advanced, industrial type of cameras that we also use in a lab setting.

These, typically, have a better lighting system and also a larger chip, so you get better images from this type of camera. Up to more professional types of cameras, where you have more advanced controls and better optics, so that you can change, for example, the lenses and give good, visual results. Additionally, another very interesting type of camera that has become much more affordable now, in recent years, is the one that we have been in the Xbox Kinect and other types of cameras, which is a depth camera. So it can sense the depth in the image, not only the plain, two dimensional image that we typically have. So these we also use a lot in music research these days.

So now we’re going to look at how it’s possible to work with video analysis on the computer. And here I have the programme Maximus P open, which is a graphical programming environment that we often use in music research. And I have different modules here for importing video from the webcamera that we have standing here now. And then I can see myself in the image. One important thing to remember when we’re working with video analysis is that a video image is, in fact, just a set of numbers. And we can look at this here by seeing these numbers change as I move. So these are numbers between 0 and 255.

When working with video analysis, one of the most common ways of starting out is to look at what we call the motion image. And the motion image is, in fact, a calculation where we’re subtracting the frames following each other in a video sequence. What that means is that whenever there is a change between two frames, that will be visible. If there’s no change, it will not be visible, as we see here now. So I have my original image here. And now I do the calculation in greyscale because it’s a little bit easier for the computer to do. Then, as we see here in the motion image, it’s only my motion that is shown and is visible.

Then I can philtre this to make it a little bit clearer and I could also add a noise removal philtre, so that we can see the motion more easily. And possibly, also, I could add some edge detection. Although, in this case, I have a very noisy background, so it doesn’t really help here. From the motion image itself, it’s possible to calculate various types of features. And one of the features that we work with a lot here, in our lab, is to look at what we call the quantity of motion, which is, in a sense, the sum of all the active pixels in a motion image. So we have the emotion image up here.

And when I move, you can see the whites pixels. If I sit still, there is nothing. And by adding up all these pixels, we get a number called the quantity of motion, which we can see here, in this blue plot here. So if I move a lot, we can see that. If I sit still, there’s not very much happening. So in many ways, the quantity of motion is also similar to the activity that we can get from inertial sensors, such as the accelerometers. And it’s a good way to look at the general movement in a sequence. We can also calculate the centroid of motion to look at where exactly in the image we are.

And also, for example, a bounding box– as we see here. That’s the square wrapped around the image– that shows the extent of the motion. These numbers, we can use for analytical purposes to plot, for example, with score material or with some sonic features. But since we also do this in real time, it’s also possible to use this to create music. For example, in this case here where I attach– [PIANO PLAYING] the horizontal motion to a piano sound. So I can play the piano by moving here. Or in another example here is with drums. [DRUM PLAYING] Where it’s possible to play drums by trying to hit different parts of the image. So you can control this in real time.

So in a way, for a lot of the things we are doing here, we can work between analysis on one side and synthesis on the other very easily because it’s really the same thing that’s going on. Now if you want to work more specifically on the analysis of, for example, larger stretches of musical movement, we need to have some tools for doing that as well. And I developed a tool– or a method I call motiongrams, which is a technique for visualising movement over time. And you could say, well, is it really necessary to visualise movement? Because movement, after all, is visual to start with.

But the thing is that just by looking at, for example, a video stream, it’s difficult to get a grasp over entire, longer sequences of movement material. So for this motiongrams as we see here, we get a plot over time of how motion evolves over time. So this is done by simply calculating the average for each of the lines in the motion image and then plotting them over time. And this can be done either horizontally or it can be done vertically, as we see here. So for this spot here, we can see the movement going up and down. While for this spot here, we can see the sideways movement.

And together, they represent the musical movement over longer stretches of time– for example, several minutes or even hours. So if I sit still, there will be nothing. And if I move, it will be displayed in these images.

Now as we talked about earlier, we’re often working between analysis and synthesis, where it’s possible to either look at things from a more scientific perspective, or from an artistic, and sometimes even in between. And again, from these motiongrams that we have looked at, it’s interesting that they can also be used to create sound. Because a motiongram is, in many ways, similar to a spectrogram of sound, so by taking a motiongram and interpreting it as a spectrogram, we can actually create sound from the video. And here is an example of this where I’m turning this motiongram into sound through an inverse FFT technique. [ELECTRONIC MUSIC NOTES PLAYING]

And even more interesting here, we can add, for example, a video effect and it will turn out to be– [HIGHER PITCHED NOTE PLAYING] An audio effect, like this blurring we are doing now, in the image here. We’ll end up with some kind of reverb with some filtering on top of it in the sound.

So these are just some examples of how it’s possible to work with video material on the computer both for analysing relationships between movement and sound, but also to create sound from movement.