The Spotlight Resonance Method: Resolving the Alignment of Embedded Activations

TL;DR

The paper introduces the Spotlight-Resonance Method (SRM) to reveal how embedded activations align with a privileged basis induced by activation functions, across layers and models. By constructing rotation operators from privileged bivectors and sweeping angular scales, SRM quantifies anisotropy in high-dimensional activation distributions and distinguishes alignments that arise from the functional form of activations. The authors demonstrate that activations tend to cluster around privileged basis directions after training, provide evidence of grandmother neurons in several networks, and show SRM's versatility across different basis constructions (elementwise, simplex, overcomplete). They argue that the observed alignment is caused by basis privileging inherent in activation functions rather than an innate property of all deep models, offering a direct causal link between symmetry breaking and representational structure with potential for guiding new activation-function designs.

Abstract

Understanding how deep learning models represent data is currently difficult due to the limited number of methodologies available. This paper demonstrates a versatile and novel visualisation tool for determining the axis alignment of embedded data at any layer in any deep learning model. In particular, it evaluates the distribution around planes defined by the network's privileged basis vectors. This method provides both an atomistic and a holistic, intuitive metric for interpreting the distribution of activations across all planes. It ensures that both positive and negative signals contribute, treating the activation vector as a whole. Depending on the application, several variations of this technique are presented, with a resolution scale hyperparameter to probe different angular scales. Using this method, multiple examples are provided that demonstrate embedded representations tend to be axis-aligned with the privileged basis. This is not necessarily the standard basis, and it is found that activation functions directly result in privileged bases. Hence, it provides a direct causal link between functional form symmetry breaking and representational alignment, explaining why representations have a tendency to align with the neuron basis. Therefore, using this method, we begin to answer the fundamental question of what causes the observed tendency of representations to align with neurons. Finally, examples of so-called grandmother neurons are found in a variety of networks.

Figures (13)

• Figure 1: Shows a representative example of combination-SRM applied to the small MNIST model with $n=24$ neurons and $m=48$ privileged basis vectors --- this creates a non-standard privileged basis along which elementwise tanh is applied. The spotlight angle used is $\epsilon=0.9$. An unseen testing data split was used to evaluate the SRM. The very faint, solid lines demonstrate the SRM fraction for each of the privileged bivectors (an ensemble line plot). Many of these translucent lines overlay, creating the dense oscillation pattern observed. The single dashed line per plot is the mean result across all privileged bivectors. The left plot shows the results of SRM for the network before training, whilst the right shows the exact same network after training. Centre shows self-SRM, which is SRM computed for the vectors of the privileged basis; this indicates what a local coding oscillation may appear like as a reference signal. It demonstrates that only after the training does the SRM oscillations become consistent with representations being axis-aligned with the privileged basis. Therefore, it appears training results in an activation symmetry breaking induced by the symmetry breaking functional forms. Due to page restrictions, further results can be found in App.\ref{['App:ExtraTests']}.
• Figure 2: The left plot demonstrates combination-SRM performed using random normal basis-bivectors, the centre shows combination-SRM using bivectors of the standard basis, whilst the right plot shows combination-SRM for the bivectors of the privileged basis. Every other parameter was kept constant across all plots, and in all cases, the number of basis vectors was equal. The network tested was the trained small MNIST model $n=10$, $m=20$ evaluated on the MNIST test set split. These results were consistent with those of other networks tested. A value of $\epsilon=0.9$ was used. The small peak on the centre plot could be for several reasons, such as a coincidently close alignment between a standard bivector and a privileged bivector or a very small subset of representations which do not display privileged basis axis alignment. In either case, the signal is very small, not oscillating or in phase with the standard basis, so can be considered insignificant.
• Figure 3: The three leftmost plots show the signed spotlight resonance method performed on handpicked single privileged bivectors. All three indicate that representations are strongly aligned with a single neuron, and the sign and strength of firing of that neuron represent a human-interpretable meaning, as discussed below. Therefore, all three are likely grandmother neurons for the CIFAR dataset. The second-to-rightmost plot is a UMAP McInnes2020 embedding of the latent layer, whilst the rightmost is a colour-coded key for the diagram. None of these observed oscillations in SRM were present before training. For the experiment, $\epsilon=0.75$ was used.
• Figure 4: The two leftmost plots show the signed spotlight resonance method performed on handpicked single privileged bivectors. Both indicate that representations are strongly aligned with a single neuron, and the sign and strength of firing of that neuron represents a human-interpretable meaning, as discussed below. Therefore, both are probably grandmother neurons. The rightmost plot is a UMAP of the embedding of MNIST in the latent layer, and it includes a colour-coded key for the diagram. None of these observed oscillations in SRM were present before training. For the experiment, $\epsilon=0.75$ was used.
• Figure 5: The left plot shows SRM performed on the untrained large CIFAR network, whilst the right plot shows SRM on the same network following training on CIFAR, and the centre plot shows the self-SRM measure. A value of $\epsilon=0.8$ was used for combination SRM --- smaller than usual. The number of basis vectors is $m=20$ in $\mathbb{R}^{10}$, therefore elementwise. Although a smaller value of $\epsilon$ was necessitated to observe any signal, a strong basis alignment can be observed. This lower value of $\epsilon$ required suggests a more diffuse alignment with the privileged basis. In the untrained plot, there are several dense crossover points in the SRM value, which can be seen at $n\pi/2$. However, these are all small valued oscillations. This is not thought to be a signal, but due to the geometry of combination-SRM, discussed further below.