Introduction to multimodality

A modality refers to one of the channels we use to get information, such as audition, vision, taste, balance and proprioception. They are the different senses we use to perceive a stimulus. In most instances, the different modalities we perceive in confirm each other: they are integrated in cognition, and strengthen our perception of a phenomenon. For example, when perceiving a car driving by, we hear its sound increase and diminish, at the same time as we see it getting closer and further away. In this case, the modalities of audition and vision confirm each other. However, in some particular cases, our mind perceives something that is not present in any of the modalities. An example of this is the McGurk effect, named after the psychologist that first described this phenomenon. You will see a demonstration of this effect in the next video.

The multimodal nature of perception may explain why we easily project features of one modality onto another. When hearing a sound on its own, we make an assumption about the look and feel of what produced it- even if we’ve never heard the sound before. What materials were used? What kind of action?

Furthermore, musical elements such as phrases, rhythmic patterns, melodies or harmonic progressions may produce associations with shapes and movement. These multimodal associations can also play a role in inducing strong emotional reactions in the perceiver. Why is this so? Move on to learn more.

References

• McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 746–748.