Advantages of Neural Population Coding for Deep Learning

TL;DR

It is shown theoretically and in experiments with synthetic data that population codes improve robustness to input noise in networks of stacked linear layers and the benefit of using population codes to encode ambiguous outputs, such as the pose of symmetric objects.

Abstract

Scalar variables, e.g., the orientation of a shape in an image, are commonly predicted using a single output neuron in a neural network. In contrast, the mammalian cortex represents variables with a population of neurons. In this population code, each neuron is most active at its preferred value and shows partial activity for other values. Here, we investigate the benefit of using a population code for the output layer of a neural network. We compare population codes against single-neuron outputs and one-hot vectors. First, we show theoretically and in experiments with synthetic data that population codes improve robustness to input noise in networks of stacked linear layers. Second, we demonstrate the benefit of using population codes to encode ambiguous outputs, such as the pose of symmetric objects. Using the T-LESS dataset of feature-less real-world objects, we show that population codes improve the accuracy of predicting 3D object orientation from image input.

Figures (8)

• Figure 1: Robustness to input perturbations comparing single-variable, one-hot vector, and population-code outputs. Here, the networks were trained without input noise. The dashed thin red line in the first panel shows the theoretical bound of $a$ below which the population-code has zero errors. Experimental results show means $\pm$ SD, as indicated by shaded areas (100 simulation runs).
• Figure 2: Robustness to input perturbations comparing single-variable, one-hot vector, and population-code outputs. Here, the networks were trained by adding Gaussian input noise. Experimental results show means $\pm$ SD, as indicated by shaded areas (100 simulation runs).
• Figure 3: Population codes lead to sparser information flow compared to one-hot vectors. The flow is the product of the neural activity times the weight of the outgoing connection. Here, the network's input was 1 for neuron with ID 10 and zero otherwise.
• Figure 4: Activation values after the last linear layer were more extreme for the one-hot approach compared to the population code. Values are mean $\pm$ SD (100 simulation runs).
• Figure 5: The population-code method showed a sparser information flow than the one-hot method. Values are mean $\pm$ SD (100 simulation runs).