Musical gesture and coarticulation

So far, we have looked at the differences between motion, action, and gesture. But this concept of gesture is quite tricky to understand. So now we’re going to visit a colleague of mine and hear how he is thinking about this concept of gesture. Come.

Hello, hello. Hi. Hi.

The baseline of all this is that music is a multimodal art. It involves both the sense of motion and the sense of sound. And, of course, it can involve also a sense of vision, even maybe the sense of smell. It’s a multimodal art. And for that reason, it’s better to try to be precise when we speak about it. When we speak about music and body motion, we can say we have body motion to make sound. We have body motion to modify sound. We have body motion that somehow complements the sound, like in dance choreography, and so on and so forth.

So it’s clear that at the end of the day, the essential point is precisely to understand how music is composed of both body motion– I prefer that term– and sound. We can now say we have a basic understanding of music as a phenomenon, well-established both from our own research and from neuroscience. that there is a very strong hardwired coupling, as they say in neuroscience, between what we hear and our sense of body motion. And as we know from our research observing musicians and dancers, most people tend to spontaneously associate musical sound with some kind of body motion. Another topic that is quite important in your research, I know, is that of coarticulation.

It’s a difficult word, but can you try to briefly explain how you’re thinking about coarticulation in music? Oh, yes. Yes, yes, with pleasure. Coarticulation means that in human motion, and by the way, also in robotics, and also in animation, you have the fact that body parts are constantly on the move. So the easiest explanation is to look at your mouth when you are speaking. Whenever you’re pronouncing a word, you can see the shape of the mouth, the lips, and the tongue. You follow the motion of the tongue. And you’ll discover that when you are saying something, you’re also preparing the next sound that you are going to make.

And also, when you’re saying something, you are, in a sense, conditioned by what you just did. In music, if you are going to play the piano and hit a key way up on the keyboard, you necessarily have to move your hand in order to hit it, because you don’t have a finger that’s that long. So this means that you always are in a context of motion. And this is quite determining for how music is shaped, both vocal music and instrumental music. So if you look at the score, what is called Western common music notations, you have dots, C-sharp, F-sharp, G, and so on, which are discrete events.

But when it comes to performance, the body has to somehow move between the keys. Or in vocal performance, you have to move from one pitch to another. So essentially, you always have a smearing. And that’s the word I use, meaning that you don’t have clean-cut different events. But they tend to go into a continuous stream of sound, exactly like in language. And that, by the way, is one of the reasons why it’s difficult to learn foreign languages, because speech is continuous. In other words, speech is coarticulated, as they say. So you have to be able to pick up the discrete events from a continuous stream. And in your research, I know you have been working theoretically on this.

But also, I know that you’re working in the lab with experiments on these things. Can you just tell a little bit about how you’re actually doing this and what you’re doing in the lab? Yes, we try to figure out exactly what musicians are doing. So far, we have, at least for my research, I’ve focused mostly on what we call sound-producing body motions in music. So what we do is that we use this so-called motion capture technology, which is essentially an infrared camera system with markers. And then we place markers on the fingers, hands, arms, shoulders, the whole torso, head to feet, and so on, depending upon what we are interested in studying.

So in the example of a piano performance, it looks like almost like a person having smallpox with all these markers on their hands. And then we have very detailed information about how this preparatory of motion is going on all the time. The next project will be drumming, a drum set. Because, as you know, you have the tom-toms, and you have the right, and hi-hat, the bass drum kicks, snare, whatever. And whatever rhythmical pattern the drummer is playing needs to have this constant motion. Because you hit the ride, and then you are hitting the snare. And the moment you hit the ride, the stick passes off, and you try to aim as best you can for the next event.

So in that sense, you always have this context. And it also is mobilising the rest of the body, the torso. The drummer would sit down on a stool like this and try as best as he or she can to exploit the rebound. A drummer has to conserve energy. Otherwise, he or she would be completely exhausted after a couple of minutes. So again, returning to your main question, you have all this embodiment, how the body is an integral part of the music-making and the body motion, of course. And then somehow, the body has to adapt to the constraints, as we say, of the physics of the musical instruments. Well, thank you very much. You’re welcome.