Parallel Backpropagation for Shared-Feature Visualization

TL;DR

This work applies a deep-learning-based approach to novel recordings from body-selective regions in macaque IT cortex in order to understand why some images of objects excite neurons in high-level visual brain regions.

Abstract

High-level visual brain regions contain subareas in which neurons appear to respond more strongly to examples of a particular semantic category, like faces or bodies, rather than objects. However, recent work has shown that while this finding holds on average, some out-of-category stimuli also activate neurons in these regions. This may be due to visual features common among the preferred class also being present in other images. Here, we propose a deep-learning-based approach for visualizing these features. For each neuron, we identify relevant visual features driving its selectivity by modelling responses to images based on latent activations of a deep neural network. Given an out-of-category image which strongly activates the neuron, our method first identifies a reference image from the preferred category yielding a similar feature activation pattern. We then backpropagate latent activations of both images to the pixel level, while enhancing the identified shared dimensions and attenuating non-shared features. The procedure highlights image regions containing shared features driving responses of the model neuron. We apply the algorithm to novel recordings from body-selective regions in macaque IT cortex in order to understand why some images of objects excite these neurons. Visualizations reveal object parts which resemble parts of a macaque body, shedding light on neural preference of these objects.

Figures (7)

• Figure 1: Why does this object image activate IT neurons that are selective for bodies? Our goal is to visually explain responses of category-selective neurons to outside-of-category (ooc.) stimuli. We start with an ooc. stimulus (object), that strongly activates a neuron which primarily fires for a specific category (bodies). We compute latent CNN activations for the image, as well as for a set of within-category reference images. A neuron-specific similarity metric operating on the latent activations finds the reference image most similar to the ooc. stimulus. The proposed parallel backpropagation method then highlights the shared features driving the neural response.
• Figure 2: Sketch of the parallel backpropagation method. A pre-trained CNN cut off at a predetermined layer computes latent feature activations. These are backpropagated to obtain the Jacobians of the two activation vectors w.r.t. to the respective images. We then calculate the normalized Hadamard product of the activation vectors and the element-wise square of the learned linear readout vector for the considered neuron. The pixel saliency map is then computed as the sum of gradients of each feature, weighted by the feature's entry in the Hadamard product.
• Figure 3: a) Correlation between predicted and recorded neural responses for held-out monkey body images (blue) and object images (orange). Significant positive correlation for object images demonstrates that the visual features predictive of responses to body images are also predictive of responses to objects. Brown dots show the correlation between responses to the same stimuli in the first and second recording phase. Dashed lines show the average across channels (colored) and the $.05$ confidence interval for the correlation coefficient under the null hypothesis that $\rho=0$ (black). b) Neural responses to the objects for which features are visualized in Figs. \ref{['fig:GradsMSB']} and \ref{['fig:Grads2']}. Responses to visualized objects are higher than mean responses to objects and bodies in the vast majority of cases.
• Figure 4: Results obtained by applying the proposed method to multi-unit recordings from body-selective cells in macaque STS (posterior region). Each subplot corresponds to one recording channel. The object on the left is among the most highly activating objects for the channel. The image on the right corresponds to the most similar preferred body.
• Figure 5: a)/b) Results for ASB region. c) Example results for objects lacking a highly similar body image among the channel's top drivers, and for weakly activating objects. Top row: strongly driving objects. Bottom row: weakly driving objects. Low similarity is properly reflected by low intensity of the saliency map.