Chapter 2 · Section 3
Vision and visual perception
From light refracted in the eye to networks that help us recognise a cup and reach for it.
Vision begins with optics. The cornea and the lens refract light so that an image of the surroundings is formed on the retina at the very back of the eye; the lens can change shape to focus at different distances. Focusing, however, is not transduction. Light is converted into electrical changes only in the retina's photoreceptors.
The rods are highly light-sensitive and dominate vision in dim light, while the cones work in daylight and provide the basis for colour vision and sharp detail vision. In the fovea, where we fixate our gaze, the cones are densely packed. The signals are processed already in the retina: the ganglion cells have receptive fields in which light in the centre and in the surround often have opposite effects, so that the cells emphasise contrasts rather than uniform illumination.
What sets the limits of what we see?
The print on a package is readable up close but turns into a grey pattern from across the room. A spinning fan is plainly visible, yet its individual blades are hard to follow. These examples raise two questions: how fine a detail can we tell apart, and how fast a change can we perceive?
Why can an object become too small to make out?
It depends not only on the object's size but also on its distance. A small letter close to the eye can cast a larger image on the retina than a large sign far away. The angle an object occupies in the visual field is called its visual angle. As the visual angle shrinks, the details of the image lie closer together on the retina.
The eye's optics do not reproduce each point as an infinitely small point. The light spreads over a small area, partly because of the wave nature of light and partly because of irregularities in how the eye refracts it. The light patterns from neighbouring details can then overlap so much that the contrast between them becomes hard to make out.
In daylight the pattern is registered mainly by the cones. How densely they are packed, and how their signals are passed on and combined by the retina's neurons, affects which details can be kept apart. Light level, contrast and further processing matter too. There is therefore no single object size that marks the limit of vision.
This is also why we see finer detail where we direct our gaze than further out in the visual field.
Explore further: The parts of the eye · The layers of the retina
Sources: Campbell (1965) · Rossi (2010)
Why can we see a star but not make out its shape?
A distant star can be very large yet occupy such a small visual angle that we cannot make out its extent. To the naked eye it then acts as a point source: we can see where the light comes from, but cannot make out the source's shape or size. Here, the word point describes what the eye can resolve, not a star with no physical extent.
Light from such a point source does not gather at an infinitely small point on the retina either. The eye's optics spread it over a small area, as in the figure below. The resulting patch of light therefore does not reveal the star's edges or surface details.
From starlight to a patch of light on the retina
1 · Light is brought together in the eye
A distant star acts as a point source. The eye's optics bring the light together, but it is distributed over a small area of the retina.
2 · Enlarged light distribution
Starlight adds light locally against the background. This can be enough to detect the star, even though its outline and surface details cannot be resolved.
We can still detect the light if enough reaches the eye and stands out from the background. Against a dark sky, the star can therefore be visible even when its extent in the visual field is far too small for us to resolve.
Detecting that something is there and making out its shape are therefore different visual tasks. The same distinction applies to seeing a single small light source in the distance, separating two nearby light sources and reading a small letter: the first does not require us to resolve the details needed for the others.
Explore further: The photoreceptors in the retina
Sources: Campbell (1965) · Hecht (1942)
Can something happen too fast for us to see it?
Yes – but here too we need to distinguish between noticing an event and making out how it unfolds. A very brief flash can be detected if enough light reaches the eye. Two flashes in quick succession, however, can be hard to perceive as two separate events.
The responses of photoreceptors and neurons are extended in time. Signals from neighbouring moments can therefore overlap and be combined. This integration over time contributes to sensitivity to light, but it also limits our ability to keep rapid changes apart.
With repeated rapid alternation between light and dark, the flicker can eventually be seen as a steady light. Where this happens depends on, among other things, the light level, the contrast and the kind of signal that is changing. It is not a fixed number of frames per second that applies to all of vision.
Explore further: Receptor potentials and nerve signals
Sources: Hecht (1942) · Kelly (1961)
Why do a fan's blades blur together when it spins?
When a fan spins quickly, the edges of its blades sweep across the same parts of the retina again and again. The image then changes so fast that the individual edges can be hard to follow, even though we clearly see that something is moving. We may see a blurred band or a translucent disc instead of each blade.
What matters includes how fast the image moves across the retina, how large the details are and how much contrast they have. The visual system's motion processing also shapes the experience: what we see is not simply a long exposure, as in a camera. Detecting motion, judging its direction and recognising details in what is moving are different tasks.
Explore further: MT in the network map · Motion and action
Sources: Burr (1997)
To understand a receptive field, follow a single ganglion cell. In the classic ON-centre example, light in the centre can increase activity while light nearby opposes that response; in an OFF-centre pathway the preference reverses. Both regions belong to the same cell's receptive field. Lateral inhibition helps explain how neighbouring signals can influence the centre through retinal circuitry. Centre–surround describes the pattern in the cell's response; the circuit connections help explain how that pattern arises.
Receptive fields and contrast enhancement
Centre–surround organisation contributes to contrast enhancement. Because light in the centre and light in the surround have opposing effects on the cell's response, differences between neighbouring parts of the visual field are emphasised. Imagine a light cup against a dark background. A receptive field near the cup's edge may receive more light in its centre than in its surround. The cell's response then provides information about a possible boundary between surfaces. It is the contrast in the nervous system's signals that is emphasised; the light from the cup does not change. Edges are one of several cues that work with colour, texture and context when we recognise an object.
When you look at the cup, each ganglion cell conveys information about part of the visual field. Many cells' receptive fields overlap, and different cell types process partly different properties in parallel. The pattern of activity across many cells therefore carries information both about where something happens in the visual field and about what changes there. An individual ganglion cell does not recognise the cup; its signals contribute to further processing.
The concept of a receptive field also applies to touch. Here it describes the area of skin that can influence a particular neuron's activity. The fields' size and their combined coverage of the skin help us understand how the nervous system can locate touch and distinguish details.
Explore one cell's receptive field →
Two models for lateral inhibition
How does the centre–surround model relate to the generic circuit in Neuron Explorer (Follow O3)? They describe the same phenomenon — activity in one part of a network can reduce a response elsewhere — but at different levels and with different calculations. The receptive-field model is a functional description of one cell: the whole centre is weighed against the whole surround in a single sum, R_ON = (centre − surround)/100, and a negative value denotes a suppressive model contribution, not a negative firing rate. The circuit instead shows, input by input, how several cells influence one cell directly and indirectly through other cells, with a signed net influence for each input. The two therefore give different numbers: the field model weighs two regions into one response, while the circuit sums distinct routes through stated connections. Neither model is a complete retina, and the circuit is a generic schematic — not a description of actual retinal wiring. You reach the field model through the link above and the circuit through the Neuron Explorer link in the visual lab.
Several routes from the same scene
Picture the cup again. Retinal receptive fields emphasise local contrasts in its contour, handle and surface. Ganglion-cell axons form the optic nerve and carry the signals onwards. A main route leads to the lateral geniculate nucleus, LGN, in the thalamus. There LGN neurons receive signals and send their own axons to the primary visual cortex, V1.
This route contains parallel P, M and K channels with differing sensitivities. P channels convey, among other things, fine spatial and chromatic contrast; M channels are especially sensitive to rapid changes and luminance contrast. K channels are heterogeneous, with some conveying short-wavelength, cone-related signals. Together these channels contribute to processing the same scene: they are not pure colour or motion pipes, and P and M do not each have a predetermined ventral or dorsal destination.
A parallel route for orienting
When something appears beside the cup, your gaze needs to be able to turn towards it. The retina also sends signals to the superior colliculus, SC, in the midbrain – not in the thalamus. An experimentally identified pathway in macaques runs from the retina through SC and the inferior pulvinar, which is thalamic, to visual area MT outside V1. It belongs to parallel visual processing and is relevant to orienting. SC is not an extra station that all signals must pass before LGN or V1, and this route is not the pulvinar's only connection.
Main route
Parallel route
Signals also travel back
Inspecting the cup's colour makes different properties important from looking for its handle. Visual cortex sends substantial feedback to LGN, and cortex and pulvinar also participate in recurrent loops. These connections help regulate processing while retinal signals continue to arrive. Attention and the task can change what is prioritised; top-down influence need not wait for a finished picture of the scene. But not every feedback signal is a conscious intention or a prediction. The thalamus is thus part of an ongoing interaction, not simply a passive relay.
Revisit feedback in section 2.2 →
Explore the connections in the existing visual pathway maps →
From retinal contrasts to cortical edges
The primary visual cortex, V1, is also called striate cortex and lies in the occipital lobe, mostly around the calcarine sulcus. It is the first cortical station of the main LGN route, not the recipient of all visual signals. Here contrasts and edge orientation are processed, and many neurons combine input from both eyes. This offers a starting point for understanding how the cup's contour, and later its depth, can be represented.
Look at the cup's contour: a border where pale china meets a darker background. For a ganglion cell with a classical centre–surround field, the response depends on how light is distributed between the centre and surround. Many such fields side by side together convey a pattern of local contrasts. The signals reach the primary visual cortex, V1, through the LGN, and many V1 neurons respond most strongly to an edge at a particular tilt.
A map in which the centre gets most room
Imagine resting your gaze on the cup's edge. Two nearby points on that edge also have neighbouring representations in the visual system. LGN projections to V1 preserve this ordering, retinotopy. Each V1 represents the opposite visual hemifield: left V1 represents the right visual field, and right V1 the left. These are halves of the visual field, not a division assigning each V1 to the opposite eye; both eyes provide information about much of the same visual field.
The map is not an undistorted photograph, however. Near the centre of gaze, a small patch of visual field gets much more cortical room than an equally sized patch far out in the periphery. This cortical magnification is an uneven distribution of processing resources, linked to finer central detail vision. Acuity also begins with the eye's optics and retinal organisation; V1 is not the sole explanation for seeing more clearly where we look.
Terms: retinotopy · cortical magnification
How is this measured?
Linear cortical magnification gives the millimetres along the cortical surface corresponding to one degree of visual angle at a particular field location: mm/degree. Visual angle describes how much of your visual field something occupies, not its size in centimetres. Areal magnification is a different measure, cortical area in mm² per square degree; distance and area should not be confused.
Human maps have been studied by comparing visual-field loss with MRI-defined lesions and through retinotopic fMRI. Duncan and Boynton measured magnification and acuity in the same observers and found a relationship. Benson and colleagues showed that map area and relative central magnification vary between people. Results therefore depend on the individual, method and visual-field coverage; a limited mapped region is not the whole of V1.
Visual field
Schematic cortical map (V1)
Region 1 is at the centre of gaze and corresponds to band 1. The centre gets most room in this qualitative map.
Qualitative scale: rings and bands show order and central expansion, not measured areas or visual angles.
Simple cells: tilt, position and light or dark
In the 1960s Hubel and Wiesel described cells – first in cats, then in monkeys – whose receptive fields have elongated, side-by-side regions where light increases the response and others where darkness does. Such a simple cell responds best to a narrow line or edge with the right orientation, in the right place and with the right contrast polarity. Rotate the line, shift it slightly or swap light for dark, and the response can disappear. Orientation here means tilt, not direction of motion.
Terms: simple cell · orientation selectivity
Complex cells: same tilt, freer position
Complex cells also have a preferred orientation, but can keep responding when the line moves within the field or when light is swapped for dark. This relative tolerance to position does not mean the response is exactly equal everywhere. Think of a vertical part of the cup's edge shifting slightly: its orientation may still suit the cell, but outside the field's sensitive regions the response decreases. “Simple” and “complex” are useful functional descriptions, not a compulsory chain that exhausts everything V1 does. The discoverers themselves cautioned that a strict hierarchy may be an oversimplification.
Terms: complex cell
End-stopped cells – classically called hypercomplex
Follow part of the cup's contour. An end-stopped cell can respond strongly to a bar of suitable length and orientation, but less when that same bar extends into the field's end zones. Contrast stays unchanged. End-stopping makes the cell sensitive to where a contour continues and ends; shortest is not always best.
Such cells were classically called hypercomplex. End-stopping is a response property that can occur in both simple and complex cells, however, not a mandatory third station. Contour endpoints and their context inform networks' processing of edges, shapes and structures, without making every cell a dedicated corner detector or switching its response off completely.
Terms: end-stopping (classically hypercomplex cells)
Three classical descriptions of receptive fields
Simple cell
Same cell · only orientation changes
Preferred vertical
Stronger response
Horizontal
Weaker response
Complex cell
Same cell · two positions within the field
Vertical, left
Response retained
Vertical, right
Response retained
End-stopped cell
Same cell · only length changes
Shorter, suitable length
Stronger response
Longer, into end zones
Reduced response
Dashed contour: a schematic receptive-field reference, not a sharp boundary. Hatched end zones show where extension can reduce the response. All bars have the same contrast. A retained response does not mean an exactly equal response.
Orientation columns and hypercolumns
Imagine a small patch around the cup's handle. In V1, many neurons have receptive fields that overlap in that part of the visual field, but prefer different orientations. In primates, neurons along a column through the cortex tend to prefer similar tilts. Moving across the cortical surface changes the preference, for example from vertical to oblique and horizontal edges. This is the basis of the term orientation column. Hubel (1974)
A hypercolumn is the classical description of a small cortical region that collectively covers different orientations for a limited part of the visual field. The model usually also includes representation from both eyes. It describes how several analyses can run in parallel for the same visual-field region; it does not mean that V1 consists of identical, sharply bounded boxes. Receptive fields overlap, and each small patch of cortex does not hold a complete picture of the object. Hubel (1968)
Terms: orientation column · hypercolumn · ocular-dominance column
Small overlapping visual-field region
Neighbours with gradually shifting preferred tilt
- 0°
- 22.5°
- 45°
- 67.5°
- 90°
- 112.5°
- 135°
- 157.5°
When signals from two eyes meet
Both eyes provide information about much of the same visual field. Along the main route through the LGN, the principal retinal inputs remain segregated in eye-specific layers. Many projections from there reach V1 layer 4C, where many neurons are driven mainly by one eye. Ocular-dominance columns describe this spatial organisation of eye preference; they are distinct from orientation columns. Hubel (1968)
Connections within and between V1 layers allow information from both eyes to contribute to a neuron's response. Many binocular neurons are sensitive to binocular disparity: the difference in where a corresponding feature falls in the two eyes' images. This provides information for stereoscopic vision. The V1 comparison is part of continuing processing; the finished depth percept cannot be equated with a single cell's response. Cumming (2000)
Terms: ocular-dominance column · binocular disparity · stereopsis
A qualification of the classical model
The classical model needs a qualification. In macaque studies, even neurons in V1's input layers driven by only one eye changed their responses when the other eye was also stimulated. “Predominantly monocular” therefore does not mean that influence from the other eye is impossible. Dougherty (2019)
Both channels concern overlapping parts of the same visual-field region, represented in one and the same hemisphere.
Two eyes Eye
LGN Thalamus
V1, layer 4C Cortex
Ocular dominance: domains driven mainly by L or R
Inputs are predominantly eye-specific, but the other eye can influence neuronal responses even here.
V1 networks, including layers above and below 4C Cortex
Many neurons receive contributions from both eyes. Sensitivity to disparity provides information for further depth processing.
The optic chiasm is omitted. The figure follows selected inputs and does not show all connections.
Orientation preference and ocular dominance are different organisational principles. Read about orientation columns and hypercolumns.
Read on: binocular depth →Explore disparity in the depth lab →
V1 in a recurrent network
V1 is not a stand-alone edge detector that sends a finished sketch upwards. It receives extensive feedback from later visual areas, and the context around a field can change its responses. As section 2.2 described, the task can change what is prioritised, without every such influence being conscious or a prediction. The edges you explore in the lab are thus an early step in interacting, parallel networks.
In V1, many neurons receive information from both eyes. Differences between the eyes' images thus become another feature for the nervous system to process. This leads to the question of how two eyes' images contribute to perceiving depth.
When visual information is used without seeing
A light appears in a part of the visual field where, following brain injury, a person reports seeing nothing. They are nevertheless asked to indicate where it appeared. If repeated responses are better than chance while the person reports no visual experience, this can be an example of blindsight. It is a limited residual ability, not ordinary vision in the affected region. Scharli (1999)
Cortical blindness means that loss of vision is caused by damage to the cerebral cortex rather than the eye or optic nerve. Each V1 represents the opposite half of the visual field from both eyes. Extensive damage to right V1 can therefore cause left homonymous hemianopia: the left visual hemifield is affected for both eyes. A smaller lesion can produce a more limited defect. Extensive damage to V1 on both sides can cause loss across the visual field. Horton (1991)
- Affected visual half-field (hatched)
- Spared visual half-field
- Fixation point
Right V1 lesion: the left half of the visual field is affected for both eyes. It is the same half of the visual world for both eyes – not blindness in one eye.
How can any visual information still be used? Not every route to visual cortex passes through V1. In people with V1 damage, preserved connections between the LGN and motion area hMT+ have been associated with residual visual ability. The route through SC and the pulvinar is also relevant to research on alternative visual pathways. The contributing connections depend on the lesion and task; blindsight has no single explanation. Ajina (2015) Schmid (2010) Berman (2010)
V1 has a central role in ordinary, detailed visual experience, but is not required for all visual processing. The boundary between absent and residual experience is not always sharp: some people can experience motion in an otherwise severely affected visual field. Blindsight occurs in some patients and must be investigated by distinguishing what a person can do from what they experience. Schmid (2010) Zeki (1998) Scharli (1999)
What does the research show?
- Scharli, H., Harman, A. M. & Hogben, J. H. (1999). Blindsight in subjects with homonymous visual field defects. Journal of Cognitive Neuroscience, 11(1), 52–66. DOI: 10.1162/089892999563247. Human lesion study separating forced-choice performance from reported awareness. Not a universal prevalence.
- Ajina, S., Pestilli, F., Rokem, A., Kennard, C. & Bridge, H. (2015). Human blindsight is mediated by an intact geniculo-extrastriate pathway. eLife, 4, e08935. DOI: 10.7554/eLife.08935. Human psychophysics and tractography; an LGN–hMT+ association, not causal proof or a universal route.
- Schmid, M. C. et al. (2010). Blindsight depends on the lateral geniculate nucleus. Nature, 466, 373–377. DOI: 10.1038/nature09179. Macaques with V1 lesions; reversible LGN inactivation abolished detection and extrastriate responses. Not a measurement of human consciousness.
- Zeki, S. & ffytche, D. H. (1998). The Riddoch syndrome: insights into the neurobiology of conscious vision. Brain, 121, 25–45. DOI: 10.1093/brain/121.1.25. The GY case: residual motion awareness after V1 lesion; distinguishes awareness from discrimination.
Read further:Several routes through the thalamusThe route through SC and pulvinarThe visual-field map in V1
Explore the SC–pulvinar route in the visual lab
Explore the LGN in the lab overview
Terms: blindsight · cortical blindness · homonymous hemianopia
Two viewpoints, one depth
Imagine sitting at the table looking at the cup. Hold a finger in front of it and, if you wish to try, keep looking at the cup while alternately closing one eye and then the other. The finger seems to shift relative to the cup. Nothing on the table needs to have moved: the eyes see the world from two different viewpoints. Alternating makes the difference easier to notice. During binocular viewing, information from both eyes is available simultaneously.
What matters is where a corresponding feature falls in each eye’s image, relative to gaze. The difference between these positions is called binocular disparity. Stereopsis is depth perception based on this disparity — not simply on any difference between two images. Different brightness or two entirely different objects do not automatically produce a shared impression of depth. The information must be comparable as parts of the same scene.
- Object point: 1.00 m · To object
- Fixation point: 1.50 m · To fixation
Depth relative to gaze
In our example, both object and fixation point lie on the midline in front of you. A point nearer than fixation gives crossed disparity; a point farther away gives uncrossed disparity. The particular midline point you fixate has zero disparity.
Now shift your gaze to the finger. The finger is now at the centre of both eyes’ views, while the cup occupies a different position relative to gaze. The objects have not moved, but the reference for comparison has changed. Vergence refers to the eye rotations that direct gaze towards the same point. Disparity, by contrast, is the difference between a feature’s image positions. Accommodation is the lens adjustment for focus. They are related in everyday viewing but are different processes.
Nearer · crossed
Object distance: 1.00 m · Fixation distance: 1.50 m
Left eye · 0.61°
Right eye · -0.61°
Calculated disparity (L − R): 1.22°
At fixation · zero
Object distance: 1.50 m · Fixation distance: 1.50 m
Left eye · 0.00°
Right eye · 0.00°
Calculated disparity (L − R): 0.00°
Farther · uncrossed
Object distance: 2.25 m · Fixation distance: 1.50 m
Left eye · -0.41°
Right eye · 0.41°
Calculated disparity (L − R): -0.81°
From image difference to interpretation
This continues the story from V1. Many binocular neurons in V1 are sensitive to particular disparities, with contributions from receptive fields for each eye. But a local image match is not the finished depth percept. Cumming and Parker recorded individual neurons in macaques: responses followed local disparity in ambiguous gratings rather than the interpreted depth order. V1 thus contributes information about how the two images relate to each other. To understand the experience of depth, we also need to follow how this information is processed in the rest of the visual system.
A row of identical edges can match in several ways. The correspondence problem is deciding which features belong together across the images; the surrounding pattern helps constrain the comparison. In human experiments with random-dot patterns, Julesz showed that disparity could yield depth without the viewer first recognising a familiar object. Depth therefore need not wait for recognition of the cup.
Depth helps us act
Stereopsis is one contribution to depth. Occlusion between objects, perspective, texture and motion parallax also provide information: with one eye, the world is not flat. In our central geometry, a given distance separation generally produces less angular disparity farther away. In experiments where people judged how a surface was slanted, Hillis and colleagues found that the contributions of texture and disparity changed with the reliability of the cues. Depth vision builds on weighing several kinds of information together.
When you choose to grasp the cup, the task guides both where you direct your gaze and which information becomes important. Where you look sets the geometry shown in the explorer; shape, position and the relations between surfaces are then processed in parallel networks with feedback. Recognising the cup and bringing your hand to the right place thus connects cooperating ventral and dorsal processing, to which we now return.
← When signals from two eyes meet in V1
Terms: Binocular disparity · Stereopsis · Vergence · Correspondence problem
Colour in changing light
Imagine the cup on the table again. By the window you see it in daylight; in the evening you switch on the desk lamp and the room fills with warmer light. The cup still looks like the same cup, with roughly the same colour. Yet something has changed physically: the light reaching your eye depends both on the surface’s reflectance and on the illumination. The lamp’s warmer light contains a different distribution of wavelengths, so the light reflected from the same surface changes. The surface is the same; the light reaching the eye is not. That the colour still feels relatively stable is one of the visual system’s most useful abilities.
Three overlapping responses
In daylight, colour vision rests on the three cone classes, S, M and L, which respond over broad and strongly overlapping wavelength ranges. Each class captures light across a large part of the spectrum, and together they cover the visible range. A single cone cannot distinguish wavelength from intensity: dim light at its most sensitive wavelength can give the same response as brighter light at a wavelength it is less sensitive to. Only when the nervous system compares the relative activity across the three classes is there a basis for telling colours apart. The cone classes are therefore not the same as the colours we name — “red” and “blue” are experiences built from comparisons, not labels on individual cones.
- S · short
- M · medium
- L · long
- Wavelength (nm), short → long
Comparisons across signals and places
The comparison continues in the retinal circuitry. A schematic description is that some cells weigh L-cone contributions against M-cone contributions, and others weigh S against the sum of L and M — in this way three overlapping responses are transformed into signals that carry colour differences onward. As with contrast, space matters: cells can compare colour between the centre and surround of their receptive field, so a surface is judged in relation to its neighbours. In the primary visual cortex (V1) of alert macaques, single-cell recordings have shown cells with double-opponent responses, sensitive to how colour is distributed across a region. Retina, LGN, V1 and later visual areas cooperate in this work; colour is processed in an interacting network rather than in any single “colour centre”.
Colour constancy: the cup keeps its colour
When the illumination shifts, the visual system receives a different input, yet the cup often keeps roughly the same perceived colour. This relative stability is called colour constancy. The visual system adjusts its sensitivity to the illumination and uses relationships between surfaces in the scene. In experiments where the illumination could be controlled precisely, people have shown clear but incomplete colour constancy. How well the colour is preserved depends, among other things, on what information about the light and the surroundings is available.
A related phenomenon is simultaneous colour contrast: the same grey patch can look different depending on the field around it, even when the patch on the screen is exactly the same. In the lab you can try both sides. In one experiment the light from the surface changes; in the other the colour patch is unchanged and only the surroundings change. Recognising the cup and reaching for it build on colour, shape and position being processed in parallel and cooperating — context is part of the processing itself, not a disturbance.
A surface does not tell the whole story
Imagine a large, uniformly coloured wall filling almost your entire field of view. Light from the wall might become more yellow because the illumination has changed, but also because the surface has a different colour. With few other surfaces to compare, it becomes harder to tell these possibilities apart.
A room with several colours offers more clues. When the illumination shifts, light from many surfaces changes together, while some relationships between their signals can remain relatively stable. The visual system can use these comparisons alongside adaptation to the illumination. We can therefore often see the cup as having the same colour even though the light from it has changed. This relative stability is called colour constancy.
Edwin Land explored this with arrangements of coloured surfaces, often called Mondrian images. With John McCann he developed Retinex theory, in which comparisons across different parts of a scene play a central role. It was an important idea: the colour we experience cannot be explained solely by the light from an isolated point. Retinex is a historical model, not a complete account of how the brain computes colour.
Later research has shown that more surfaces can, under certain conditions, help us judge changes in illumination. But neither the immediate surround, the scene’s average colour nor its brightest surface alone is enough to explain colour constancy. Several processes work together, and constancy is often incomplete. More contrasts therefore do not guarantee that the influence of illumination disappears.
Look at the two small squares. They have exactly the same screen colour, but different surroundings. Switch between blue–green and yellow–red neighbours and compare how the right-hand square looks. Then hide the surroundings.
Against blue–green neighbours the sample may look warmer; against yellow–red neighbours it may look cooler. The surroundings change colour. The sample itself changes only when you adjust the illumination.
The two samples have exactly the same physical screen colour in every state. Adjusting the illumination changes all surfaces together. Hiding the surroundings does not change the samples.
Here we explore how surroundings influence colour appearance. Colour constancy, by contrast, means that a surface can retain roughly the same perceived colour when the illumination changes. The phenomena are related, but this comparison does not by itself show how colour constancy arises.
The strength of the effect varies between observers and displays. The illumination is a simplified screen illustration, and the colours are not individually brightness-matched. The images do not reproduce Land’s original experiments.
Research background: Land & McCann (1971) · Linnell & Foster (2002) · Kraft & Brainard (1999) · Gordon & Shapley (2006)
Terms: The three cone classes (S, M, L) · Opponent processing · Colour constancy · Chromatic adaptation · Simultaneous colour contrast
From an edge to a cup
You have followed how V1 processes local contrasts and the orientation of edges. To perceive the cup, we also need to distinguish it from its background, follow its shape and notice whether it moves. V2, V3 and V4 contribute to this work together with other visual areas. We will explore some of their contributions using the same cup.
Finding edges and contours is a useful strategy for recognising objects, but it is one of several. The brain also uses colour, surface texture, motion and the context in which an object appears. The cup's outline offers one clue, its glaze another, and its place on a set table a further one. These cues can work together and complement one another, especially when part of the object is hidden or hard to make out.
V2: which surface does the edge belong to?
Follow the contour where the cup meets the background. A small patch of edge tells us about a light–dark difference, but is not enough to determine which side belongs to the cup. That requires information about how the contour continues and the surrounding surfaces. Assigning an edge to a particular figure is called border ownership.
Recordings from macaque visual cortex show that many V2 neurons respond differently to the same local edge depending on which side contains the surrounding figure. Information outside the classical receptive field therefore affects the response. Such responses also occur in other visual areas, including V4. V2 helps us understand how local edges can be related to a larger figure; a single cell's response is not the same as recognising the cup. Zhou (2000)
The surroundings are visible: the same local edge sits in two different contexts.
Terms: border ownership
V3: when a contour changes position
Now the cup is pushed across the table. Its vertical edge can keep the same tilt while the whole cup moves to the right. Orientation and motion direction describe different properties: how the edge is tilted and which way it moves. The visual system needs to use both, together with depth and other information.
In macaques, researchers have found V3 neurons sensitive to orientation, motion direction and differences between the eyes' images. Studies have also shown overlap between colour and motion sensitivity. V3 thus provides a clear example of form, motion and colour being processed in interacting systems. The cup in the figure helps us distinguish these properties; its movement should not be read as a simulation of what V3 does on its own. Felleman (1987) Gegenfurtner (1997)
What does the research show?
- Felleman, D. J. & Van Essen, D. C. (1987). Receptive field properties of neurons in area V3 of macaque monkey extrastriate cortex. Journal of Neurophysiology, 57(4), 889–920. DOI: 10.1152/jn.1987.57.4.889. Recordings from anaesthetised macaque V3 (as delineated at the time): selectivity for orientation, motion direction and binocular disparity.
- Gegenfurtner, K. R., Kiper, D. C. & Levitt, J. B. (1997). Functional properties of neurons in macaque area V3. Journal of Neurophysiology, 77(4), 1906–1923. DOI: 10.1152/jn.1997.77.4.1906. Macaque V3: overlap between colour and motion sensitivity, including direction-selective responses to isoluminant stimuli.
V3 and the neighbouring V3A are distinct areas. The single-cell results here come from macaques and should not automatically be assigned to every part of human V3. Boundaries and names differ between maps.
Terms: motion direction
V4: the handle's curve and the cup's colour
Let your gaze follow the handle. A single orientation is not enough to describe it: the contour curves, and several edge segments together form a shape. Recordings in macaques show that many studied V4 neurons are sensitive to contour features such as curves and angles and the directions they point. This provides information for further shape processing and recognition in temporal networks and elsewhere. Pasupathy (1999)
The cup's colour supplies further information about the same object. V4 contributes to both shape and colour processing, but colour vision relies on cooperation across several areas, starting in the retina. In the figure you can change the colour while keeping the contour, then change the handle's shape without changing the colour. The properties can be described separately while belonging to the same cup.
What you are looking for also matters. When you search for the handle, its contour matters; when comparing two glazes, colour differences become more relevant. In macaque experiments, attention changed V4 neuronal responses even though stimulation in their receptive fields stayed the same. Attention therefore does not need to introduce a new object into the image to influence how information is processed. McAdams (1999)
Change the handle shape or the surface colour; the other property stays the same.
What does the research show?
- Pasupathy, A. & Connor, C. E. (1999). Responses to contour features in macaque area V4. Journal of Neurophysiology, 82, 2490–2502. DOI: 10.1152/jn.1999.82.5.2490. Macaque V4: selectivity for the curvature and angles of contour parts – a basis for shape processing, not whole-cup recognition in one cell.
- McAdams, C. J. & Maunsell, J. H. R. (1999). Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4. Journal of Neuroscience, 19(1), 431–441. DOI: 10.1523/JNEUROSCI.19-01-00431.1999. Macaque V4: attending to a stimulus inside the receptive field changed response strength without systematically sharpening orientation tuning.
The contour and attention studies concern macaque V4. Human hV4 is mapped using other methods and has partly different boundaries; shared names do not imply identical maps.
When colour disappears from experience
The cup can retain its outline while its colour is no longer experienced. In cerebral achromatopsia, brain damage causes a profound loss of colour vision. In severe cases, people describe a world in shades of grey; others have a partial impairment. Lesions often involve posterior ventral occipital and occipitotemporal regions. Other aspects of vision can be impaired at the same time. Bouvier (2006)
Colour and form are different properties
With colour information
Without colour information (greyscale)
V4 first attracted attention for its colour-sensitive neurons in macaques. Later experiments showed that background illumination can change the responses of colour-sensitive V4 cells in ways consistent with changes in colour perception. This supports a contribution to colour constancy. A cell's activity is not itself a colour experience, however, and human colour vision cannot be localised to one colour module. Kusunoki (2006)
This brings us back to the handle and glaze: V4 contributes to several properties of the same object, including colour and contour shape. Colour loss after brain injury shows that some parts of processing are especially important for colour vision, not that the rest of the visual system operates without colour information. Pasupathy (1999) Bouvier (2006)
What does the research show?
- Bouvier, S. E. & Engel, S. A. (2006). Behavioral deficits and cortical damage loci in cerebral achromatopsia. Cerebral Cortex, 16(2), 183–191. DOI: 10.1093/cercor/bhi096. Lesion-overlap analysis of 92 published cases; variable severity, coexisting spatial impairments, ventral occipital overlap. Not isolated V4 lesions or a pure colour-only module.
- Kusunoki, M., Moutoussis, K. & Zeki, S. (2006). Effect of background colors on the tuning of color-selective cells in monkey area V4. Journal of Neurophysiology, 95(5), 3047–3059. DOI: 10.1152/jn.00597.2005. Macaque ventral V4 recordings with background illumination changes, compared with psychophysics. Modulation related to colour induction/constancy; no claim that cells themselves perceive.
Read further:Colour and colour constancy
Try lighting and surroundings in the colour lab
Terms: cerebral achromatopsia
Several processes at once — with feedback
The names are easy to read as a work sequence: first V1, then V2, V3 and V4. But the visual system works through multiple concurrent and recurrent connections. Information about the cup's edges, colour and motion is processed in overlapping networks. An area sends signals onwards while also being influenced by signals from other areas.
Feedback carries influences back to earlier processing stages. When researchers temporarily reduced activity in macaque MT/V5, responses in V1, V2 and V3 changed, particularly for figures that were difficult to distinguish from a background. This shows that early processing is influenced by what happens elsewhere in the network. Hupé (1998)
Top-down describes how factors including task, attention and prior knowledge can influence processing. Looking for the handle and judging the cup's colour place different demands on the system even when the image is the same. Feedback is part of this interaction, but not every feedback signal expresses a conscious intention. The parallel streams continue to work together as our task changes. McAdams (1999) Ruff (2008)
For this task, the handle's position and contour matter most – shape and location information in cooperating networks. Colour is still processed, but it does not decide the answer.
Select an area for a short description and a link to the visual lab's map.
Connections in the figure
- V1 → V2 · Forward projection
- V2 → V3 · Forward projection
- V2 → hV4 · Forward projection
- V3 → MT / hMT+ · Forward projection
- hV4 → IT · Route via several areas
- V2 → V1 · Feedback
- hV4 → V2 · Feedback
- MT / hMT+ → V1 · Feedback
V1 · occipital lobe
V1, the primary visual cortex, lies in the occipital lobe. Here the main pathway through the LGN reaches the cerebral cortex for the first time. V1 is retinotopically organised, receives LGN projections in layer 4C and superficial layers, and sends feedback back to the LGN.
A selection of connections, mainly from macaque studies – not the complete anatomy and not a measured human wiring diagram.
IT: how do you recognise the cup?
The cup now stands on another part of the table. It occupies less of your visual field, and part of its handle may be hidden. Yet you can often still recognise it as a cup. Object recognition requires us to distinguish objects while also recognising them when their images change.
Inferior temporal cortex, IT, lies in the lower parts of the temporal lobe and is part of ventral visual processing. Activity patterns here carry information about objects and categories. Comparisons of macaque neuronal responses and human fMRI patterns have revealed similarities in how different objects are represented. The account concerns patterns across many neurons; it does not require us to assume a single dedicated “cup cell”. Kriegeskorte (2008)
Builds on:V4: the handle's curve and the cup's colourColour and colour constancySeveral processes at once
See IT in the map of visual pathways →
Several clues to the same object
Shape tells you about the cup's body, opening and handle. Colour and surface texture offer other clues: the glaze may be blue or terracotta, plain or speckled. Two cups can therefore have similar shapes but different appearances. And the same kind of object can be recognised even when its colour or surface pattern differs.
Human studies show that shape, colour and texture are processed in partly different regions of occipitotemporal networks. We therefore need to follow several contributions to recognition. IT participates in a wider interaction; the entire object need not first be assembled in one single region where all properties meet. Cavina-Pratesi (2010)
Try changing one property at a time in the picture. Notice what changes and what still makes the object a cup for you. Then change the setting without changing the cup image itself. Which clues do you use now? The example shows the information available in the image; it does not measure how your brain uses it.
Shape
Colour
Surface texture
Setting
The picture shows – Shape: rounded · Colour: terracotta · Surface texture: plain glaze · Setting: breakfast table.
What remained recognisable, and what changed?
The same object, different images
When an image of the cup moves or becomes smaller, many local details in the visual field change. In macaque experiments, IT activity patterns distinguished objects across several tested image changes better than V4 patterns did. This tolerance helps explain how recognition can persist as an image varies. Tolerance is limited: it does not mean every IT neuron responds identically regardless of position, size or viewpoint. Rust (2010)
Near, in the centre
Farther away and smaller, on the left
Moved to the right
The setting helps you interpret
A cup among breakfast plates fits a familiar setting. The same cup among tools on a workbench suggests different expectations, even if the cup image is unchanged. In experiments with briefly presented photographs, people identified objects more accurately when the object fitted its setting than when it did not. This shows that context can influence recognition; it does not locate the effect solely in IT or mean that every change of setting changes what you see. Davenport (2004)
What you are trying to do matters too. When looking for your blue cup, colour is useful; when reaching for it, the handle's shape and position matter. Task, prior experience and feedback influence processing in interacting networks. Recognition emerges from this interaction between available visual information and the context in which you use it. McAdams (1999) Ruff (2008)
Explore task and feedback in the map →
What does the research show?
- Kriegeskorte, N. et al. (2008). Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6), 1126–1141. Human fMRI compared with macaque single-neuron populations: similar categorical structure in IT. The methods measure different things, and the similarity concerns broader IT, not every subdivision.
- Cavina-Pratesi, C., Kentridge, R. W., Heywood, C. A. & Milner, A. D. (2010). Separate channels for processing form, texture, and color: evidence from fMRI adaptation and visual object agnosia. Cerebral Cortex, 20(10), 2319–2332. DOI: 10.1093/cercor/bhp298. Humans: fMRI adaptation and two patients with visual object agnosia. Form, texture and colour were processed partly separately in occipitotemporal regions — not everything in a single node.
- Rust, N. C. & DiCarlo, J. J. (2010). Selectivity and tolerance (“invariance”) both increase as visual information propagates from cortical area V4 to IT. Journal of Neuroscience, 30(39), 12978–12995. DOI: 10.1523/JNEUROSCI.0179-10.2010. Macaques: V4 and IT populations were compared across tested image transformations. Both selectivity and tolerance increased from V4 to IT; tolerance was limited, not perfect single-neuron invariance.
- Davenport, J. L. & Potter, M. C. (2004). Scene consistency in object and background perception. Psychological Science, 15(8), 559–564. DOI: 10.1111/j.0956-7976.2004.00719.x. Humans: briefly presented, masked photographs (80 ms) with objects in consistent or inconsistent settings. A behavioural result, not a localisation to IT or a prediction for every scene you view without time pressure.
- McAdams, C. J. & Maunsell, J. H. R. (1999). Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4. Journal of Neuroscience, 19(1), 431–441. DOI: 10.1523/JNEUROSCI.19-01-00431.1999. Macaque V4: attention changed response strength. Effects of attention and context are not confined to IT.
- Ruff, C. C. et al. (2008). Distinct causal influences of parietal versus frontal areas on human visual cortex: evidence from concurrent TMS–fMRI. Cerebral Cortex, 18(4), 817–827. Humans, TMS and fMRI: influence from parietal networks on visual areas. Does not establish a specific feedback synapse.
Terms: inferior temporal cortex (IT) · tolerance · top-down
Seeing something and recognising it
Imagine the cup clearly in front of you, yet you cannot identify what it is. In visual agnosia, recognition through vision is impaired in a way that cannot be adequately explained by poor visual acuity, for example. The difficulty varies: some people struggle to organise the form, others to connect what they see with what the object means. This develops the question we followed through IT: how do multiple visual cues become something familiar? Goodale (1991) Cavina-Pratesi (2010)
Read further:IT and recognising the cup
A face without recognition
In prosopagnosia, or face blindness, a person can perceive a face yet struggle to recognise whose face it is. Acquired prosopagnosia can follow damage to face-processing networks, often involving an important contribution from right ventral occipitotemporal cortex. The fusiform face area, FFA, is one part of this network; the lesion need not lie within FFA itself to impair recognition. Cohen (2019)
There is also developmental prosopagnosia, in which difficulties have been present from early life without an acquired brain injury explaining them. Face-recognition ability varies across people, and estimates of how many have prosopagnosia depend on the tests and diagnostic cutoffs used. DeGutis (2023)
How common is it?
The often quoted figure of about two per cent is an approximate estimate. DeGutis and colleagues examined how different diagnostic criteria produced different prevalence estimates in the same large sample. DeGutis (2023)
Read further:Faces, motion and social cues
When written words become difficult to read
Recognising a cup and quickly reading the word ‘cup’ place partly different demands on the visual system. In some forms of acquired alexia, reading becomes severely impaired after brain injury. An important region lies in left ventral occipitotemporal cortex and is commonly called the visual word form area, VWFA. It works with language networks; it is not a separate store of word meanings. Gaillard (2006)
What does the research show?
- Goodale, M. A., Milner, A. D., Jakobson, L. S. & Carey, D. P. (1991). A neurological dissociation between perceiving objects and grasping them. Nature, 349, 154–156. DOI: 10.1038/349154a0. The DF case: visual form agnosia and a perception–action dissociation in the tested tasks; not independent, universally spared streams.
- Cohen, A. L., Soussand, L., Corrow, S. L., Martinaud, O., Barton, J. J. S. & Fox, M. D. (2019). Looking beyond the face area: lesion network mapping of prosopagnosia. Brain, 142(12), 3975–3990. DOI: 10.1093/brain/awz332. Human lesion-network study; a network connected to right FFA, lesions need not overlap FFA. Normative connectivity, not monosynaptic anatomical wiring.
- DeGutis, J. et al. (2023). What is the prevalence of developmental prosopagnosia? An empirical assessment of different diagnostic cutoffs. Cortex, 161, 51–64. DOI: 10.1016/j.cortex.2022.12.014. Prevalence varies with definition and cutoffs; about 2% is not universal. The lifelong developmental form is distinguished from acquired injury.
- Gaillard, R. et al. (2006). Direct intracranial, fMRI, and lesion evidence for the causal role of left inferotemporal cortex in reading. Neuron, 50(2), 191–204. DOI: 10.1016/j.neuron.2006.03.031. Single human case before and after surgery with multiple methods; selective acquired reading deficit after resection of left word-responsive occipitotemporal cortex. Not all reading disorders arise from one locus.
Terms: visual agnosia · prosopagnosia · acquired alexia
After V1, processing is distributed across many visual areas that work in parallel while also building on one another. A ventral processing towards the temporal lobe emphasises recognition of shapes, objects and faces. A dorsal processing towards the parietal lobe emphasises where things are and how we direct actions towards them. The division describes emphases, not two isolated pipes: the streams exchange information, and there is extensive feedback from later to earlier stages, all the way back to the LGN.
When we look at another person, body movements, gaze and expressions also matter. Regions around pSTS contribute to this social perception in interaction with other visual areas. We can therefore follow three interacting emphases: recognition, spatial processing and action, and interpretation of social cues. In the demonstration below, you can begin with biological motion. Go to the biological-motion demonstration →
The task can change what is prioritised within these coexisting pathways. Finding the red cup makes different features relevant from judging whether it is tipping; attention and feedback influence processing without excluding an entire stream. Revisit task relevance in section 2.2 →
Recognising the cup and grasping it thus rely on partly different processing – but in everyday life both happen at the same time and affect each other. Choose a stage in the overview below to read more about it.
- Eye
- The retina lines the inner back of the eye.
- Below the cerebral cortex
- The LGN lies in the thalamus, deep within the brain.
- Cerebral cortex
- Via the LGN, the main pathway first reaches V1 in the occipital lobe.
Retina – in the eye
The retina is a thin layer of neural tissue lining the inner back of the eye. Photoreceptors convert light into electrical changes. The retina's network compares signals in ON and OFF pathways, and the ganglion cells send action potentials via the optic nerve.
Explore the visual pathways – connection maps for P/M/K, colliculus and pulvinar, cortical areas and feedback.
Following movement and guiding the hand
When the cup slides across the table, you need to follow its movement. In akinetopsia, motion perception is severely impaired. A classic patient with damage to posterior brain regions on both sides showed how movement could become difficult to perceive while other visual abilities were better preserved. Motion-sensitive networks, which include MT/V5, should therefore be distinguished from the networks that use vision to guide the hand. Zihl (1983)
Following the cup's movement
Motion perception. Can be severely impaired in akinetopsia; motion-sensitive networks including MT/V5 contribute.
Guiding the hand to the cup
Visually guided action. Can be disrupted in optic ataxia; posterior parietal networks contribute.
In optic ataxia, a person can see an object yet misdirect the hand when reaching for it or fail to adjust the hand to its orientation. Damage to posterior parietal regions can disrupt this transformation from visual information into action. This differs from failing to recognise the cup or perceive its movement. These examples help explain the different emphases of ventral and dorsal processing while the streams still need to cooperate. Perenin (1988)
What does the research show?
- Zihl, J., von Cramon, D. & Mai, N. (1983). Selective disturbance of movement vision after bilateral brain damage. Brain, 106, 313–340. DOI: 10.1093/brain/106.2.313. The LM case: bilateral posterior damage, motion impairment with some remaining motion function. Lesion extent does not isolate MT alone.
- Perenin, M. T. & Vighetto, A. (1988). Optic ataxia: a specific disruption in visuomotor mechanisms. I. Different aspects of the deficit in reaching for objects. Brain, 111, 643–674. DOI: 10.1093/brain/111.3.643. Ten patients with unilateral parietal damage; reaching and hand-orientation errors that can dissociate from visuospatial perception. Not equivalent to akinetopsia or hemianopia.
Read further:Cooperating visual streams
Explore parietal networks in the visual lab
Terms: akinetopsia · optic ataxia
When movement tells us about a person
Someone across the table reaches for the cup. You see the hand, arm and upper body moving together. These coordinated patterns help us perceive a person's action. This is an example of biological motion. Even when the body is shown only as a few points of light at the joints, their movement can be enough for us to recognise a person walking. Johansson (1973)
When points of light become a person
Start the motion and compare the two displays. In one, the points follow a simplified walking body. In the other, each point has been moved to a new location but keeps its own motion. What happens to the overall impression?
Both displays show 15 white points on a dark background, moving in synchrony at the same rate. On the left, the points sit where the head, shoulders, elbows, wrists, hips, knees, ankles and toes would be on a person walking in place. On the right, each point makes exactly the same movement, but from a different fixed location in the display.
Schematic walking in place
An original schematic animation, not a recording of a person or a test of emotion recognition.
A single point shows only a small movement. When the points' positions and movements fit together, they can form a pattern we perceive as a moving body. The comparison helps distinguish each point's motion from the body's overall organisation. Some movements may still provide clues in the scrambled display; the demonstration need not produce exactly the same experience for everyone.
Regions around the posterior superior temporal sulcus, pSTS, contribute to our perception of biological motion. They lie in the cerebral cortex on the outer side of the temporal lobe. Changing social cues, such as gaze direction and facial expressions, are also processed here. STS therefore matters when we try to understand what another person is doing and what their expressions may mean. Grossman (2005)
Movement can also provide clues to emotion. A sudden step backwards or a changing facial expression can influence how we perceive another person's reaction. Studies that temporarily disrupted activity in right pSTS with magnetic stimulation found impaired recognition of emotional expressions, in both faces and body movements. STS works together with other regions. A movement needs to be interpreted in context; it does not reveal with certainty what a person feels. Pitcher (2014) Basil (2017)
What does the research show?
- Grossman, E. D., Battelli, L. & Pascual-Leone, A. (2005). Repetitive TMS over posterior STS disrupts perception of biological motion. Vision Research, 45(22), 2847–2853. DOI: 10.1016/j.visres.2005.05.027. Human TMS: temporary disruption over posterior STS impaired the perception of biological motion. Supports a contribution of the region to this perception.
- Pitcher, D. (2014). Facial Expression Recognition Takes Longer in the Posterior Superior Temporal Sulcus than in the Occipital Face Area. Journal of Neuroscience, 34(27), 9173–9177. DOI: 10.1523/JNEUROSCI.5038-13.2014. Human TMS over right pSTS impaired the matching of facial expressions in still photographs, but not the matched identity task. The experiment did not use moving faces.
- Basil, R. A., Westwater, M. L., Wiener, M. & Thompson, J. C. (2017). A Causal Role of the Right Superior Temporal Sulcus in Emotion Recognition From Biological Motion. Open Mind, 2(1), 26–36. DOI: 10.1162/opmi_a_00015. Human cTBS with individually fMRI-localised right pSTS (23 participants): impaired recognition of emotion from point-light body movements, while a non-biological global-motion task was unaffected. Evidence for a contribution – not an isolated 'emotion centre'.
The paragraph above summarises these separate studies together; they were not a single combined experiment and do not necessarily involve exactly the same neurons.
Explore pSTS in the visual lab →
Back to the retina-to-cortex overview →
From movement to emotional meaning
Seeing that someone is walking is one thing. Perceiving them as happy, anxious or about to avoid something requires an interpretation of the expression and the situation. Here vision research meets affective neuroscience, which studies how the nervous system contributes to emotions and their interaction with perception, the body and action. We also need to distinguish the emotion we attribute to the person from our own response to what we see.
Body movements can carry information about emotional expression even when the face is not visible. In experiments using actors' expressions, participants could identify intended emotions from point-light movements. These findings show that movement patterns can provide useful cues. They do not mean that a particular speed always signifies a particular emotion, or that we can know with certainty what someone is experiencing internally. Atkinson (2004)
Processing engages several networks. pSTS contributes to perceiving movements and expressions, while emotional significance is also studied in regions including the amygdala, insula and orbitofrontal cortex. In an fMRI study, images of fearful body postures produced greater activity than neutral body postures in several of these regions and in regions associated with action. The study used still images. It shows a pattern of co-occurring activity, not the order in which signals travel or that any one region creates an emotion on its own. de Gelder (2004)
The same movement, different contexts
Keep the movement unchanged and choose an imagined situation. Does the story influence how you interpret the person? This is a thought comparison: there is no correct emotion label, and no effect is being measured.
You see someone walking. What can the movement alone tell you?
Context can influence how an emotional expression is perceived. In experiments with intense facial expressions, judgements changed when the same face was combined with different bodily expressions. Our thought comparison using a story illustrates the question of context; it does not reproduce that experiment. Expectations can influence interpretation, but the movement alone does not determine which story is true. Aviezer (2012)
What does research show about movement and emotion?
- Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception & Psychophysics, 14, 201–211. DOI: 10.3758/BF03212378. Original point-light study (behaviour): a few lights at the joints were enough to perceive human movement. Our 15-point animation is an original schematic, not Johansson's stimuli, motion capture or a replication of the experiments.
- Atkinson, A. P., Dittrich, W. H., Gemmell, A. J. & Young, A. W. (2004). Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception, 33(6), 717–746. DOI: 10.1068/p5096. Behavioural experiments: actors portrayed emotions with faces covered; participants classified them in forced-choice tasks across static and dynamic, full-light and point-light displays. Shows that intended expressions can be recognised, not what someone feels inside, and does not validate our gait or speeds.
- de Gelder, B., Snyder, J., Greve, D., Gerard, G. & Hadjikhani, N. (2004). Fear fosters flight: A mechanism for fear contagion when perceiving emotion expressed by a whole body. PNAS, 101(47), 16701–16706. DOI: 10.1073/pnas.0407042101. Human fMRI with still whole-body images and faces blanked: fearful versus meaningful neutral postures produced co-occurring activity in affect-related, visual and action-related regions. An association – not a directed chain, necessity or a single emotion centre.
- Aviezer, H., Trope, Y. & Todorov, A. (2012). Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 338(6111), 1225–1229. DOI: 10.1126/science.1224313. Behavioural experiments with intense facial expressions and face–body composites: the body changed how the facial emotion was judged. Our thought comparison with the same gait and different stories is not that paradigm and demonstrates no measured effect.
Open the visual lab Explore the eye, receptive fields and visual pathways. You will return here.
Social cues across interacting networks
Recognising who someone is and perceiving where they are looking are different tasks. Describing STS as part of a social visual stream emphasises the processing of gaze, expression and actions. It is a useful way to follow information, not a third isolated channel alongside the other two. Experiments temporarily disrupting pSTS show contributions to specific tasks; they do not place all understanding of other people in that region. Grossman (2005) Pitcher (2014) Basil (2017)
When the person begins to speak, vision and hearing meet. Lip movements can influence the speech sound we perceive. Magnetic-stimulation experiments support a contribution of parts of STS to this interaction, including the McGurk effect. This provides a natural connection to hearing: what we hear and see can together shape our experience of the same event. Beauchamp (2010)
STS and variation in social perception
An fMRI study of autism found group differences in how STS activity was affected by the context of another person's gaze shift. Such findings need to be tied to the task and participants studied. They do not show that autism is caused by damage to STS or that all autistic people perceive social cues in the same way. Pelphrey (2005)
Human task-fMRI group comparison of context modulation during gaze shifts. An association, not causal focal-lesion evidence, a diagnostic biomarker or a universal feature of autistic people.
Explore pSTS in the visual lab
Read about how vision and hearing interact