The mobile mocap studio

Hello. In this video, we will look at mobile technologies for studying music and body motion. In the laboratory setting, we are able to use highly advanced infrared optical motion capture systems to study very precisely how people move to music. But sometimes, we need to go outside the lab and study how people move in the real world. How people move while listening to music on the metro, on the bus, or in the dance club. For this reason, we need to have some sort of mobile technology for studying music and body motion. In front of me, I have a couple of examples of such technologies.

One of the advantages of this system is that it can be very, very small, such as this accelerometer. Imagine studying body motion in a dance club, where you can put this in the pocket of every dancer in the club. And you get some data on how they move, and can see how this relates to the music and how they compare to each other. Accelerometers and gyroscopes, and also magnetometers, are inside basically every mobile phone these days. And so everybody, in practise, has their own motion capture technology that they carry around. More specialised systems also exist, such as this bracelet which contains sensors that measure muscle tension.

In addition to this accelerometer and gyroscope that lets you measure the direction of the bracelet, and also how fast it accelerates. Like this. So it will tell you which direction I point and how I turn my hand, because of the muscle sensors.

This suit is a motion capture suit from Xsens Technologies. It contains 17 of these orange sensors. And inside each of these there is a magnetometer, and a gyroscope, and accelerometer, which measures inertia, and acceleration, and the magnetic flux. Or basically, a compass. Combining 17 of these sensors with a model of a human body, saying that, for instance, this sensor and this sensor can move further apart than this. And this sensor and this sensor can only move in this angle. It’s possible to get an estimation of how the pose of the person is at any time. And the result you get is quite good, actually, from thinking that this type of technology there are no cameras involved.

The system is portable, with two senders like this. And it’s possible to move quite far away from the transmitter, which is right next to our computer. So looking at the screen now, my hand is upright and the avatar is following my movements. We tested it up to 50 metres without problems.

One small problem with this system is that it’s drifting. So if you move around a lot and go back to the same place, you will possibly have a result that diverges slightly. And also, it’s not able to give you a millimetre position like you’ll get in the optical systems. Still, it’s remarkably good and absolutely a good way of measuring body movements. What you can see here is that there’s quite a lot of cables. So using it in, for instance, a stage performance requires that you are aware of your system, and you will not be able to move as freely as without the system.

But we have used it in several performances, and you can use it as a DJ to control DJ sets or as a musical instrument.