Capturing a string quartet

OK. In the last section, we looked at the details of how we can do motion capture. How the cameras work, the calibration, and also some of the post-processing. In this section, we’re going to look at some examples of how this type of technology is used in real research. And we’re going to then talk to some of the researchers about their experiments. And we’ll start out by looking at some examples of motion capture of musicians. You can do this by looking at individuals or by looking at multiple performers. Laura, you have experience with doing this.

And in the case of what we did in the library with a string quartet– this was actually the second time we recorded this string quartet, because we already recorded them once before, here in the lab. Yes. What, in your experience, is the difference between doing such a recording in the lab and doing it in a public setting? Well, in a public setting, it requires a lot of on-the-spot preparation. In the lab, you have your cameras in place already, so you can kind of prepare in advance where the musicians are going to sit. And you know the cameras are going to be in a useful position, and you’ve got everything connected.

But when you try to capture something in real life, like we did in the library, basically an hour or two before the musicians come, you have to put all the cameras in a useful place and try to point them so that they’re facing the direction that you expect the musicians are going to be. This is not always very easy or very successful. And you might have to deal with some difficulties with the lighting, or reflections coming from– if the floor is not ideal and kind of shiny, then this could also be a problem. So it requires a lot of on-the-spot troubleshooting. But the quality of the data is more or less the same, or what do you think?

Yeah well actually, in both of these cases, it was kind of a trial run. So in neither case we actually had the ability to set up very much in advance and have the musicians in advance, so we could make sure that everything was going to be captured well. So the quality of the data ended up being similar in both cases. We had some issues with the backs of the second violinist and the violist not being captured well, because they ended up being placed a little bit too closely to the edge of the MoCap space.

And so, for instance, in the library concert, now that we’ve seen this data and seen that this is the problem, next time we would know that we need to put cameras lower down, right behind the people. Maybe move the musicians forward a little bit. And I guess in both of these settings, we actually had also an audience involved, right? So even in the lab setting we decided to run it with an audience, and why was that? Well in this case, we were interested in how the musicians’ behaviour differs between whether there’s an audience or not. So when there’s an audience, there’s more pressure, it’s more stressful. It’s also more exciting.

And so we were testing some hypotheses about how the musicians might respond to these conditions. And in particular, I was kind of expecting that they might be more interactive, more overtly interactive with each other. So in that case, we would see certain differences in the way they move, the amount that they move, and how their movements relate to each other. And you have some graphs here as well. So what do we see here? This is the trajectory of the marker that was placed on the back of the cello in the vertical direction, over the course of a performance of about two minutes. So this is actually measured in terms of the distance from the starting position.

So you can see how much she changed her position over the course of the performance. Is that a type of– do you often like to use that type of measure, instead of looking at the position? Actually I don’t usually use this kind of measure. In this particular case, I used this measure because I was making a comparison of two different motion capture systems that were used to capture the same performance session. And because the cameras are in different positions than the actually absolute position values that we got for the markers were in different coordinate systems, and these had to be compared somehow.

So taking the position as a distance from the starting position from the origin of the marker was a useful way to make that comparison. And then, since in this case, you actually did then study and use two different motion capture systems, what is your experience with the systems and how similar are they? They actually are very similar. It was quite interesting, because we had– it was an interesting comparison. Because we had one very, very expensive top-of-the-line system, which was about 10 years old at the time that we made the capture. And then an almost brand new system, which was– so the cameras are a lot smaller and more affordable, but was brand new.

So it was actually very interesting to see that the systems performed almost the same. Really there was not too much difference. Normally, you would expect the high end system to perform better. But this is a 10-year-old system. And then, in terms of the noise, then that you can see in the systems, that’s something that we have been interested in here because we also have studies where we look at smaller movements. What do we see here? This, in red and blue, this is the noise that we found in a small clip of a recording from a marker that was standing still.

So this marker was actually placed on one of the music stands of one of the performers, and so it should not be moving at all. But even markers that are sitting perfectly still register some movement in the system. And this is what we refer to as noise. And this is unavoidable. So this graph shows the noise for the same marker from the two systems. And if you look at the y-axis, this is in millimetres, so this is fractions of a millimetre. So it’s really actually a very, very small, number. Very, very tiny amount of movement. So even this is kind of big jump is not actually a big jump. This is 0.2 of a millimetre.

This is interesting, because it shows the different kinds of noise that you can see in these systems. So you get a lot of this very, very small up and down jitter. And then sometimes you get these kind of spikes that are just jumps that jump back. And then you also get this kind of thing, where the whole marker position shifts a very small amounts. It says in the same place and then goes back to where it was. So I guess this is something that we often do in our studies, is that we like to have a still marker to be able to also measure the noise level of the system.

It’s good as a reference, I think, for these type of things. Yes. And then, if you were kind of trying to summarise all your experience with working with capturing musicians or pianists and string players, et cetera, what are kind of the most important things do you think to think about when you do that type of motion capture study? I guess you have to think very carefully about the type of motion that you want to capture. The parts of the body that you want to capture. And come to an agreement with yourself about which parts of the body you want to measure, and which parts are actually going to be measured successfully.

So for instance, with the string musicians, you see that some of the body is covered by the instrument. So for instance, with pianists, maybe it’s interesting to look at the shoulders. With a string musician, like a violinist, who’s got their violin here, you can’t really get this motion. And you can’t really get the arm motion. So if it’s important for you to have this, you need to choose whether it’s important for you to have this, and then maybe you need to place the markers in a specific way. So then, what you’re saying is that the marker placement is very important for these type of things. And particularly to also to consider the instrument. Yes. Yes?

Yes, and it’s also interesting to measure the movement of the instrument in many cases. I guess this is a bit philosophical, but to some extent, the instrument becomes part of the person. It’s being controlled by the person, and for some musicians, this may be a kind of part of themselves as they’re producing the music. So in that way, it’s very interesting to look at how they are manipulating the instrument and the bow, if there’s a bow involved. OK. So that’s some good tips when it comes to capturing musicians. And we’re going to then jump over to looking at also how we can capture perceivers.