Skip main navigation

Introduction to multimodality

Multimodality in music perception. How is it that we associate a sound with shape and movement?
© University of Oslo

Multimodality is a term used to emphasise that our senses work together.

A modality refers to one of the channels we use to get information, such as audition, vision, taste, balance and proprioception. They are the different senses we use to perceive a stimulus. In most instances, the different modalities we perceive in confirm each other: they are integrated in cognition, and strengthen our perception of a phenomenon. For example, when perceiving a car driving by, we hear its sound increase and diminish, at the same time as we see it getting closer and further away. In this case, the modalities of audition and vision confirm each other. However, in some particular cases, our mind perceives something that is not present in any of the modalities. An example of this is the McGurk effect, named after the psychologist that first described this phenomenon. You will see a demonstration of this effect in the next video.

The multimodal nature of perception may explain why we easily project features of one modality onto another. When hearing a sound on its own, we make an assumption about the look and feel of what produced it- even if we’ve never heard the sound before. What materials were used? What kind of action?

Furthermore, musical elements such as phrases, rhythmic patterns, melodies or harmonic progressions may produce associations with shapes and movement. These multimodal associations can also play a role in inducing strong emotional reactions in the perceiver. Why is this so? Move on to learn more.


© University of Oslo
This article is from the free online

Music Moves: Why Does Music Make You Move?

Created by
FutureLearn - Learning For Life

Reach your personal and professional goals

Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.

Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.

Start Learning now