Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks

TL;DR

This study investigates how local topographic regularization shapes robustness and internal representations in end-to-end trained CNNs. By contrasting Weight Similarity (WS) and Activation Similarity (AS) losses, the authors show WS yields smooth, functionally localized maps and greater robustness to weight perturbations, while AS produces non-smooth, striped activation patterns and distinct representational changes. Both regularizers improve robustness relative to non-topographic controls, yet they reshape the latent space differently, influencing activation entropy, dimensionality, and the distribution of category-selective expert units. The work demonstrates that topographic constraints can enhance robustness without necessarily sacrificing performance and provides a framework for understanding how cortical-like spatial organization emerges from learning, with implications for pruning, compression, and future extensions to larger architectures and unsupervised objectives.

Abstract

Topographic convolutional neural networks (TCNNs) are computational models that can simulate aspects of the brain's spatial and functional organization. However, it is unclear whether and how different types of topographic regularization shape robustness, representational structure, and functional organization during end-to-end training. We address this question by comparing TCNNs trained with two local spatial losses applied to a penultimate-layer topographic grid: i) Weight Similarity (WS), whose objective penalizes differences between neighboring units' incoming weight vectors, and ii) Activation Similarity (AS), whose objective penalizes differences between neighboring units' activation patterns over stimuli. We evaluate the trained models on classification accuracy, robustness to weight perturbations and input degradation, the spatial organization of learned representations, and development of category-selective "expert units" in the penultimate layer. Both losses changed inter-unit correlation structure, but in qualitatively different ways. WS produced smooth topographies, with correlated neighborhoods. In contrast, AS produced a bimodal inter-unit correlation structure that lacked spatial smoothness. AS and WS training increased robustness relative to control (non-topographic) models: AS improved robustness to image degradation on CIFAR-10, WS did so on MNIST, and both improved robustness to weight perturbations. WS was also associated with greater input sensitivity at the unit level and stronger functional localization. In addition, as compared to control models, both AS and WS produced differences in orientation tuning, symmetry sensitivity, and eccentricity profiles of units. Together, these results show that local topographic regularization can improve robustness during end-to-end training while systematically reshaping representational structure.

Figures (26)

• Figure 1: Overview of main concepts. Objects ($O_1$--$O_3$) are passed to a neural network. Two example output units ($\star = u$ and $\blacktriangle = v$) are indicated. For each example unit, an activation vector ($\mathbf{a}_u, \mathbf{a}_v$) summarizes that unit’s responses across objects, and a weight vector ($\mathbf{w}_u, \mathbf{w}_v$) summarizes the incoming weights from the preceding layer. Activation similarity is computed by correlating activation vectors across objects. Weight similarity is computed as a distance between weight vectors.
• Figure 2: Test accuracy as a function of topographic regularization strength $\bm{\lambda}$. Mean classification accuracy for control (dashed), AS (blue), and WS (orange) models is shown for the different values of $\lambda$. Error bars indicate $\pm$ s.e.m.
• Figure 3: Robustness to weight perturbations as a function of noise levels. Robustness is evaluated by changes in representational geometry (top row) and test accuracy (bottom row) for increasing levels of additive weight noise. Representational geometry is defined as the representational similarity matrix (RSM) computed from class weight vectors (CWVs) in the readout layer. Cosine similarity between original and perturbed representations is shown in the top row; corresponding changes in test accuracy relative to baseline (non-perturbed) models are shown in the bottom row. Shaded regions indicate min--max range across $\lambda$ levels. Values computed separately for each $\lambda$ level and model are reported in Supplementary Figure \ref{['fig:Approbweight']}.
• Figure 4: Model-group robustness under input corruptions. For each dataset and corruption type (white, pink, and salt-and-pepper), the most robust model group is defined as the one showing the smallest drop in test accuracy relative to the uncorrupted baseline. On MNIST, WS models are most often the most robust; on CIFAR-10, AS models are generally most robust. Each cell reports the $\lambda$ value associated with the winning group. A full breakdown of accuracy drops across $\lambda$ and corruption intensities is reported in Supplementary Figure \ref{['fig:Appnoise_acc']}.
• Figure 5: Unit-level entropy and Percentage-of-Zero activations as a function of topographic regularization strength ($\lambda$). The two left plots show average entropy of pre-ReLU unit activations for MNIST and CIFAR-10 across values of $\lambda$. The two right plots show the average Percentage-of-Zero (PoZ) of post-ReLU unit activations. Error bars indicate $\pm$ s.e.m.