Local vs distributed representations: What is the right basis for interpretability?

TL;DR

The results highlight that distributed representations constitute a superior basis for interpretability, underscoring a need for the field to move beyond the interpretation of local neural codes in favor of sparsely distributed ones.

Abstract

Much of the research on the interpretability of deep neural networks has focused on studying the visual features that maximally activate individual neurons. However, recent work has cast doubts on the usefulness of such local representations for understanding the behavior of deep neural networks because individual neurons tend to respond to multiple unrelated visual patterns, a phenomenon referred to as "superposition". A promising alternative to disentangle these complex patterns is learning sparsely distributed vector representations from entire network layers, as the resulting basis vectors seemingly encode single identifiable visual patterns consistently. Thus, one would expect the resulting code to align better with human perceivable visual patterns, but supporting evidence remains, at best, anecdotal. To fill this gap, we conducted three large-scale psychophysics experiments collected from a pool of 560 participants. Our findings provide (i) strong evidence that features obtained from sparse distributed representations are easier to interpret by human observers and (ii) that this effect is more pronounced in the deepest layers of a neural network. Complementary analyses also reveal that (iii) features derived from sparse distributed representations contribute more to the model's decision. Overall, our results highlight that distributed representations constitute a superior basis for interpretability, underscoring a need for the field to move beyond the interpretation of local neural codes in favor of sparsely distributed ones.

Figures (11)

• Figure 1: (a)$\bullet$Local (neuron) versus $\bullet$Distributed (sparsely distributed vector) visual representations. The activation of individual neurons may be driven by multiple unrelated visual elements (depicted in the images at the bottom) whereas distributed representations, obtained via dictionary learning methods, break down complex patterns into simpler ones corresponding to single visual features. (b) In practice, dictionary learning methods "disentangle" local activations to yield a new vector basis whose activation is driven by single features. The hope for interpretability is that those features align better with the set of features that humans can interpret $S = \{f_1, f_2, ..., f_n\}$.
• Figure 2: Illustration of a trial. Example of a trial in our study corresponding to Experiment I, distributed representation condition of a unit located in $layer 2.0.bn2$. Two panels of 9 reference images are located on the left and right-hand side of the display, separated by 2 query images in the center. Participants were asked to select the query image they believed shared the same visual elements as the reference images displayed on the right panel, corresponding to maximally activating stimuli. The less ambiguous this shared visual element is—the more visually coherent the set of images—the more likely participants are to select the correct query. In this case, the correct query is the bottom image depicting a yellow tram.
• Figure 3: Illustration of the role of semantics. Example of a trial from Experiment I in the local representation condition. In this case, the task can be trivially solved by simply relying on semantics. By observation of the minimally activating stimuli (left panel), it is easy to conclude that the neuron of interest is not a monkey detector, yet, it is hard to articulate what is the visual feature captured by the neuron (images in the right panel).
• Figure 4: Per-layer results for Experiment I (a) and II (b). Given a feature and a set of images to illustrate it, we assess how visually coherent participants find this set of images—or how unambiguous the feature is. More precisely, we measure the proportion of participants that are able to identify the query image which is also part of this set of images. In both experiments, a clear trend emerges where features appear significantly less ambiguous in the distributed representation than in the local representation condition, particularly in the deeper layers of the network.
• Figure 5: Feature importance. We measure the importance of a feature as the average drop in logit score $\Delta y$ for the 300 most activating images when the feature is occluded. Except for $layer 1$, we find that the model relies significantly more on features derived from the distributed representation than on features from local representations, $z=-5.86$, $p<.001$ (Mann-Whitney U test).