Chapter 2 · Section 5

When the senses work together

In a conversation you see the mouth move and hear the words. What you perceive often builds on both.

Imagine a conversation in a noisy café. You hear the voice, but you also see the mouth move. What you perceive as the spoken words often builds on both sources. Seeing the mouth movements can make speech easier to follow when the sound is hard to hear – an example of multimodal perception.

A well-known example is the McGurk effect: when a video shows the mouth movements for one syllable while the soundtrack contains another, many people report hearing a third, fused syllable. The strength of the effect varies between people, languages and stimulus materials – and it cannot be recreated with text on a page, because it requires real sound and real video. The point is that the visual impression can influence what one experiences hearing.

Figure 2.6. Vision and hearing in a conversation. Mouth movements and speech sounds reach the listener through different senses and together contribute to the interpretation of the speech. How much they are combined depends on whether they plausibly belong to the same event.Original schematic illustration.

The senses' information is weighted together depending on how well the signals fit. The combination requires no deliberate effort, but it is not unconditional: signals that coincide in time, come from the same place and fit together in content are combined more readily. More senses therefore do not always give better precision: if the signals conflict, the result can be an illusion or a confusion. The same principle applies to the body. When you reach for the cup, what you see of the hand is combined with touch and proprioception – the sense of the joints' and muscles' positions – so that you know where the hand is and when you touch the handle.