Primary visual cortex contributes to color constancy by predicting rather than discounting the illuminant: evidence from a computational study

TL;DR

The paper investigates color constancy and the neural role of V1, challenging the view that V1 DO neurons discount the illuminant. It builds an electrophysiology-inspired V1 network with variants that include subunit processing, divisive normalization, and surround effects, trained to predict the scene illuminant from natural images. Results show learned RFs resemble simple and double-opponent DO neurons, with DO cells providing more robust illuminant prediction; DO-only or simple-cell-only networks can perform comparably, suggesting DO cells encode illuminant information. The work supports an illuminant-encoding role for DO cells within an efficient coding framework and offers guidance for both neuroscience and bio-inspired computer vision.

Abstract

Color constancy (CC) is an important ability of the human visual system to stably perceive the colors of objects despite considerable changes in the color of the light illuminating them. While increasing evidence from the field of neuroscience supports that multiple levels of the visual system contribute to the realization of CC, how the primary visual cortex (V1) plays role in CC is not fully resolved. In specific, double-opponent (DO) neurons in V1 have been thought to contribute to realizing a degree of CC, but the computational mechanism is not clear. We build an electrophysiologically based V1 neural model to learn the color of the light source from a natural image dataset with the ground truth illuminants as the labels. Based on the qualitative and quantitative analysis of the responsive properties of the learned model neurons, we found that both the spatial structures and color weights of the receptive fields of the learned model neurons are quite similar to those of the simple and DO neurons recorded in V1. Computationally, DO cells perform more robustly than the simple cells in V1 for illuminant prediction. Therefore, this work provides computational evidence supporting that V1 DO neurons serve to realize color constancy by encoding the illuminant,which is contradictory to the common hypothesis that V1 contributes to CC by discounting the illuminant using its DO cells. This evidence is expected to not only help resolve the visual mechanisms of CC, but also provide inspiration to develop more effective computer vision models.

Figures (11)

• Figure 1: Comparison of four simulated V1 neural models on predicting illuminant. The left is the average of the median angular error of 60 results on the whole Cube+ dataset. The right is the average of the mean angular error of 60 results on the whole Cube+ dataset. Each error bar indicates the standard error of the mean. The p values labelled above the bar are obtained using the pairwise Wilcoxon signed rank test on 60 results of the prediction accuracy on three validation folds. Similar interpretation applies to the results on the REC dataset in the second row, in which the statistics are obtained on 120 results of the prediction accuracy on three validation folds.
• Figure 2: Trained convolution kernels during the iteration of the epochs. The random weights of the kernel gradually converge to a color-opponency subunit structure (a) and (b) red-green color-opponency RF, and (c) blue-yellow color-opponency RF.
• Figure 3: Responses of DN for several examples of indoor and outdoor scenes.
• Figure 4: Trained convolution kernels during the iteration of 20 epochs under conditions of predicting the illuminant (the first row) and discounting the illuminant (the second row). Images are displayed for every other epoch.
• Figure 5: Samples of RFs of V1 neural models learned from the IP task. The learned RFs can be classified into several types, such as localized and oriented, which are quite similar to the simple, $DO_{LM-opponent}$, and $DO_{S-sensitive}$ RFs in V1 neurons recorded using the spike-triggered average (STA) method de2021spatial. Some RFs in the dotted line box show clear color-opponency structure (see the descriptions in the main text). The recorded data are adapted from Figure 3 and Figure 4 in De and Horwitz de2021spatial.