Skip main navigation

Introduction to multimodality

Multimodality in music perception. How is it that we associate a sound with shape and movement?
© University of Oslo

Multimodality is a term used to emphasise that our senses work together.

A modality refers to one of the channels we use to get information, such as audition, vision, taste, balance and proprioception. They are the different senses we use to perceive a stimulus. In most instances, the different modalities we perceive in confirm each other: they are integrated in cognition, and strengthen our perception of a phenomenon. For example, when perceiving a car driving by, we hear its sound increase and diminish, at the same time as we see it getting closer and further away. In this case, the modalities of audition and vision confirm each other. However, in some particular cases, our mind perceives something that is not present in any of the modalities. An example of this is the McGurk effect, named after the psychologist that first described this phenomenon. You will see a demonstration of this effect in the next video.

The multimodal nature of perception may explain why we easily project features of one modality onto another. When hearing a sound on its own, we make an assumption about the look and feel of what produced it- even if we’ve never heard the sound before. What materials were used? What kind of action?

Furthermore, musical elements such as phrases, rhythmic patterns, melodies or harmonic progressions may produce associations with shapes and movement. These multimodal associations can also play a role in inducing strong emotional reactions in the perceiver. Why is this so? Move on to learn more.

References

© University of Oslo
This article is from the free online

Music Moves: Why Does Music Make You Move?

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education