Filling-in

In vision, filling-in phenomena are those responsible for the completion of missing information across the physiological blind spot, and across natural and artificial scotomata. There is also evidence for similar mechanisms of completion in normal visual analysis. Classical demonstrations of perceptual filling-in involve filling in at the blind spot in monocular vision, and images stabilized on the retina either by means of special lenses, or under certain conditions of steady fixation. For example, naturally in monocular vision at the physiological blind spot, the percept is not a hole in the visual field, but the content is “filled-in” based on information from the surrounding visual field. When a textured stimulus is presented centered on but extending beyond the region of the blind spot, a continuous texture is perceived. This partially inferred percept is paradoxically considered more reliable than a percept based on external input. (Ehinger et al. 2017).

A second type of example relates to entirely stabilized stimuli. Their colour and lightness fade until they are no longer seen and the area fills in with the colour and lightness of the surrounding region. A famous example of fading under steady fixation is Troxler's fading. When steadily fixating on the central dot for many seconds, the peripheral annulus will fade and will be replaced by the colour or texture of the background. Since the adapted region is actively filled-in with background colour or texture, the phenomenon cannot be fully explained by local processes such as adaptation.

There is general agreement that edges play a central role in determining the apparent colour and lightness of surfaces through similar filling-in mechanisms. However, the way in which their influence is performed is still unclear. Two different theories have been put forward to explain the filling-in completion phenomenon.

One theory, addressed as the "isomorphic filling-in theory" according to the definition of Von der Heydt, Friedman et al. (2003), postulates that perception is based on an image representation held in a two dimensional array of neurons, typically arranged retinotopically, in which colour signals spread in all directions except across borders formed by contour activity. The process is thought to be analogous to physical diffusion, with contours acting as diffusion barriers for the colour and brightness signals. An alternative hypothesis is that image information is transformed at the cortical level into an oriented feature representation. Form and colour would be derived at a subsequent stage, not as the result of an isomorphic filling-in process, but as an attribute of an object or proto-object. This theory is called the symbolic filling-in theory.

According to the isomorphic filling-in theory, colour is represented by the activity of cells whose receptive fields point at the surface, but it is assumed that these cells receive additional activation through horizontal connections that keeps their activity level high despite mechanisms of lateral inhibition tending to suppress surface activity and despite the transient nature of the afferent signals. The lateral activation comes from receptive fields at contrast borders. These signals are strong because receptive fields are exposed to contrast, and reliable because the border produces continuous light modulation even during fixation, due to small residual eye movements. In the alternative symbolic hypothesis, there is no spreading of activity, but all the information would be carried by the relevant features, that would be tagged with information on contrast polarity, colour and lightness of the surfaces they enclose. Despite the many attempts to verify the two different models by psychophysical and physiological experiments, the mechanisms of colour and lightness filling-in are still debated.

Isomorphic filling-in

There are at least three different kinds of experiments whose results support the idea that a real spreading of neural activity in early visual areas is the basis for filling-in of visual information.

Recordings from cells of the blind spot representation in monkey striate cortex

Komatsu and colleagues (Komatsu et al., 2000) recorded activity of cells of the blind spot representation in monkey striate cortex (area V1) and found some cells, in layers 4–6, that responded to large stimuli covering the blind spot (the condition under which filling-in is perceived), but not to small stimuli near the blind spot. A neuronal circuitry seems to exist that elaborates and transmits colour and brightness information through the blind region.

Though intriguing, these results cannot be easily generalized to similar phenomena, such as the filling-in of illusory contours or the filling-in through artificial scotomata or adapted edges (such as in the Troxler's effect). All these phenomena are indeed similar, and probably rely on similar neural circuitries but they are not identical. For instance, an obvious difference between filling-in across the blind spot and filling-in of occluded edges is that filling-in across the blind spot is modal (i.e. you literally see the filled-in section), while filling-in across occluders is amodal. Filling-in across the blind spot was found to be different also from filling-in across cortical scotomata in two patients examined by V. S. Ramachandran (Ramachandran 1992; Ramachandran, Gregory et al. 1993). In these subjects, some features filled in the scotoma faster than others, and in some circumstances filling-in took some seconds before it was completed (while filling-in across the blind spot is immediate). Together these data suggested that mechanisms for the filling-in of colours, motion and texture can be dissociated and may correspond to processes in higher-order areas that are specialized for these attributes.

Delayed masking psychophysical experiments

Striking evidence implying a spreading of neural activity like the one postulated by isomorphic filling-in theory is given by experiments of backward masking after brief presentations of uniform surfaces or textures. The working hypothesis of these experiments is that if a response initially biased toward the boundaries fills-in to represent the interiors of uniform surfaces, it may be possible to interfere with the filling-in process and leave the percept at an incomplete stage.

Paradiso and Nakayama (1991) performed an experiment to verify this hypothesis. They presented a large disk of uniform brightness on a black background. The stimulus was briefly flashed and, after a variable stimulus offset asynchrony, a masking stimulus was presented. The mask consisted of a circle on a black background with the masking contours positioned within the boundaries of the large uniform disk. This experiment is grounded on the assumption that filling-in consists of a spreading of neural activity from the boundaries of luminance and through the surfaces, that is stopped when another luminance-contrast border is reached (this is proposed by many models of brightness perception, see for example Walls 1954, Gerrits and Vendrik 1970, Cohen and Grossberg 1984), and that the process takes some time to be completed.

Subjects were asked to match the brightness at the centre of the disk with a palette of grey scales. When the delay between target and mask presentation was long enough, the mask had no effect on the apparent brightness of the stimulus, but for stimulus offset asynchronies of 50–100 ms, the surface of the disk inside the masking annulus appeared unfilled. Moreover, the minimum target-mask delay at which the masking was effective increased with target size, suggesting that there would be a spreading phenomenon and that the farther the features delimiting a region, the more time is necessary for the filling-in to be completed. These results are supported also by further experiments on temporal limits of brightness induction in simultaneous contrast (De Valois, Webster et al. 1986; Rossi and Paradiso 1996; Rossi, Rittenhouse et al. 1996), as well as by a similar experiment performed by Motoyoshi (1999) on filling-in of texture.

Recordings from striate cortex of cat

An isomorphic filling-in theory calls for the existence of surface responsive neurons in early retinotopic visual areas. The activity of such neurons would be raised by elements capable of responding to the luminance of the surface also in the absence of edges; and would be strongly modulated by spreading of activity from the luminance borders enclosing the surface.

Electro-physiological recordings in retinal ganglion cells, LGN and primary visual cortex showed that neurons of these areas responded to luminance modulation within the receptive field even in the absence of contrast borders.

In a second condition, a uniform grey patch was placed on the receptive field (extending 3–5 degrees beyond the receptive field boundary on either side), and two flanking patches modulated sinusoidally in time from dark to light. With such stimuli, the brightness of the central patch appears to modulate, despite the absence of luminance change. In this condition, cat retinal ganglion cells and lateral geniculate nucleus cells, having their receptive fields centred in the uniform grey patch, did not respond; on the other hand, primary visual cortex neurons were modulated by luminance changes far outside their receptive fields. Together, these results suggest that neurons in the retina and LGN are responsive to luminance modulation, but their response does not correlate with perceived brightness. On the other hand, striate neurons responded to stimulus conditions producing changes in brightness in the area corresponding to the receptive field.

The behaviour of primary visual cortex neurons seems to be in agreement with the one hypothesized by an isomorphic filling-in theory in that they both respond to luminance of the surfaces also in the absence of borders, and their activity is modulated by that of edges far outside the receptive field. Moreover, when the temporal frequency of luminance modulation in the surrounding patches exceeded a threshold value, the induced response disappeared, suggesting that it was the result of a spreading of activity, taking a finite time to happen, likely explainable in the context of isomorphic filling-in.

Symbolic filling-in

"Perceptual filling-in", in its simplest definition, is simply the filling-in of information that is not directly given to the sensory input. The missing information is inferred or extrapolated from visual data acquired in a different part of the visual field. Examples of filling-in phenomena include lightness assignment to surfaces from information of contrast across the edges and completion of features and textures across the blind spot, based on the features and textures that are detected in the visible part of the image. In this definition, it is clear that a filling-in process involves a rearrangement of visual information, in which activity in one region of the visual field (i.e. edges) is assigned to other regions (surfaces). In any event, the total amount of information available is not increased, being determined by the retinal input, and any rearrangement of information is useful only if it brings the information contained in the image into a form that is more easily analyzed by our brain.

Dennett and Kinsbourne (Dennett 1992; Dennett and Kinsbourne 1992) opposed to the idea that an active filling-in process would take place in our brain on philosophical grounds. They argued that such an idea would be the result of the false belief that in our brain there is a spectator, a sort of homunculus similar to ourselves, needing a filled-in image representation. From a scientific viewpoint, Dennett's homunculus may correspond to higher-order scene representation or decision-making mechanisms. The question is whether or not such mechanisms need a filled in, gap free representation of the image to function optimally (Ramachandran 2003).

The symbolic filling-in theory postulates that such a "homunculus" need not exist, and that image information is transformed at the cortical level into an oriented feature representation. Surface form and colour are not coded at this stage, but would be derived only at a symbolic level of representation, as attributes of objects or proto-objects.

Electrophysiology recordings from monkey primary visual cortex

When experiencing perceptual filling-in, the colour of an adapted surface is gradually replaced by the colour or texture of outside the surface. Friedman et al. (Friedman 1998; Friedman, Zhou et al. 1999) performed an experiment aimed at determining if surface activity of cells in monkey primary visual cortex changed in accordance with perceptual change or simply followed the modulation of the colour presented to the retina. Stimuli consisted of a disk-ring configuration similar to that illustrating the Troxler effect, but where the inner and outer part of the annulus have two physically different colours. After a few seconds of (peripheral) fixation, the disk tends to disappear, whereas the outer contour of the ring is perceived much longer, and the area of the disk is filled-in with the colour of the ring (Krauskopf 1967). These stimuli where intermixed with control stimuli, in which the physical colour of the disk was gradually changed to that of the ring. The animals were instructed to signal a colour change, and their responses to control stimuli and to test stimuli were compared in order to determine if monkeys perceive colour filling-in under steady fixation like humans.

The authors recorded the activity of surface- and edge-cells (cells whose receptive fields pointed either to the filled-in surface or to the border between the disk and the ring) in the visual cortices V1 and V2 while the monkey was performing the filling-in task. The activity of surface-cells correlated with the physical stimulus change in both areas V1 and V2, but not with the perceived colour change induced by filling-in. The activity of edge-cells followed the stimulus contrast when the disk colour changed physically; when the colours were constant, the edge signals also decayed, but more slowly. Together, the data was incompatible with the isomorphic filling-in theory, which assumes that colour signals spread from the borders into uniform regions.

fMRI experiments in human subjects

The neuronal activity in different brain areas can be recorded in humans through non-invasive techniques, like fMRI (functional magnetic resonance imaging). Perna et al. (2005) used fMRI to investigate the neuronal mechanisms responsible for the Craik–O'Brien–Cornsweet illusion. These authors recorded the activity in different brain areas when observers were presented with a Cornsweet visual stimulus, and compared the activities with those elicited by a similar image, which however did not elicit any brightness filling-in.

Contrary to the predictions of isomorphic filling-in, these authors found an identical response to the stimulus that induced filling-in and to the control stimulus in early visual cortex.

Recently, Cornelissen et al. (2006) performed a similar experiment involving the simultaneous contrast illusion. These authors presented observers in an fMRI scan with simultaneous contrast stimuli composed of a central circle of uniform luminance and a peripheral region whose luminance was modulated in time (and also tested other conditions where the modulated and the constant regions were inverted). The brain activity was recorded in primary visual cortex in a retinotopic position corresponding to the perceptually filled-in region. Also in this condition, no activity was found at this level in response to the filled-in signal.

References

Ehinger, B.V.; Häusser, K.; Ossandon, J.P.; König, P. (2017). "Humans treat unreliable filled-in percepts as more real than veridical ones". eLife. 6: e21761. doi:10.7554/eLife.21761. PMC 5433845. PMID 28506359.
Cohen, M.A.; Grossberg, S. (1984). "Neural dynamics of brightness perception: features, boundaries, diffusion, and resonance". Percept Psychophys. 36 (5): 428–56. doi:10.3758/bf03207497. PMID 6398424.
Cornelissen, F.W.; Wade, A.R.; et al. (2006). "No functional magnetic resonance imaging evidence for brightness and color filling-in in early human visual cortex". J Neurosci. 26 (14): 3634–41. doi:10.1523/jneurosci.4382-05.2006. PMC 6674117. PMID 16597716.
Dennett, D. (1992). "Filling in" versus finding out: a ubiquitous confusion in cognitive science. Cognition: Conceptual and Methodological Issues. v. d. B. P. Pick HL Jr, Knill DC. Washington DC, American Psychological Association: 33–49.
Dennett, D.C.; Kinsbourne, M. (1992). "Time and the observer – the where and when of consciousness in the brain". Behav Brain Sci. 15 (2): 183–201. doi:10.1017/s0140525x00068229. S2CID 15053574.
Gerrits, H.J.; Vendrik, A.J. (1970). "Simultaneous contrast, filling-in process and information processing in man's visual system". Exp Brain Res. 11 (4): 411–30. doi:10.1007/bf00237914. PMID 5496938. S2CID 19515051.
Komatsu, H.; Kinoshita, M.; et al. (2000). "Neural responses in the retinotopic representation of the blind spot in the macaque V1 to stimuli for perceptual filling-in". J Neurosci. 20 (24): 9310–9. doi:10.1523/JNEUROSCI.20-24-09310.2000. PMC 6773023. PMID 11125010.
Komatsu, H (2006). "The neural mechanisms of perceptual filling-in". Nat Rev Neurosci. 7 (3): 220–31. doi:10.1038/nrn1869. PMID 16495943. S2CID 2665270.
Motoyoshi, I (1999). "Texture filling-in and texture segregation revealed by transient masking". Vision Res. 39 (7): 1285–91. doi:10.1016/s0042-6989(98)00248-x. PMID 10343842.
Paradiso, M.A.; Nakayama, K. (1991). "Brightness perception and filling-in". Vision Res. 31 (7–8): 1221–36. doi:10.1016/0042-6989(91)90047-9. PMID 1891814. S2CID 25547992.
Perna, A.; Tosetti, M.; et al. (2005). "Neuronal mechanisms for illusory brightness perception in humans". Neuron. 47 (5): 645–51. doi:10.1016/j.neuron.2005.07.012. PMID 16129395.
Ramachandran, V.S.; Gregory, R.L. (1991). "Perceptual filling in of artificially induced scotomas in human vision". Nature. 350 (6320): 699–702. Bibcode:1991Natur.350..699R. doi:10.1038/350699a0. PMID 2023631. S2CID 4281067.
Ramachandran, V.S.; Gregory, R.L.; et al. (1993). "Perceptual fading of visual texture borders". Vision Res. 33 (5–6): 717–21. doi:10.1016/0042-6989(93)90191-x. PMID 8351843. S2CID 21298287.
von der Heydt, R., H.S. Friedman, et al. (2003). Searching for the Neural Mechanisms of Color Filling-In. Filling-In: From Perceptual Completion to Cortical Reorganization. P. De Weerd. Oxford, Oxford University Press: 106–127.
Sacks, Oliver (2010) The Mind's Eye Knopf, reviewed, New York Times Book Review (11 Nov 2010) Sacks' own filling-in experience with his scotoma.
Tononi, Fabio, “Ernst Gombrich and the Concept of ‘Ill-Defined Area’: Perception and Filling-In”, Journal of Art Historiography, 29: 2 (2023), pp. 1–27.
Troxler, D. (1805). Ueber das Verschwindern gegebener Gegenstande innerhalb unsers Gesichtskreises. Ophtalmologisches Bibliothek. J. Schmidt. Jena, Springer: 431–573.