Barycentric alignment for instance-level comparison of neural representations

TL;DR

The paper tackles the problem of comparing neural representations across diverse models in the presence of symmetry-induced ambiguities. It introduces a barycentric alignment framework that quotients nuisance transformations via a Procrustes-based barycenter to map multiple models into a single universal embedding space, enabling instance-level similarity assessments. Across vision, language, cross-modal, and brain data, the approach reveals stimulus-specific convergence and divergence, shows that post-hoc alignment of unimodal models can yield cross-modal similarity aligned with human judgments, and highlights differences between artificial and biological representational variability. These findings demonstrate that much of inter-model representational structure is geometry-driven and recoverable through symmetry-preserving alignment, with implications for understanding universality, cross-domain transfer, and brain-model correspondence.

Abstract

Comparing representations across neural networks is challenging because representations admit symmetries, such as arbitrary reordering of units or rotations of activation space, that obscure underlying equivalence between models. We introduce a barycentric alignment framework that quotients out these nuisance symmetries to construct a universal embedding space across many models. Unlike existing similarity measures, which summarize relationships over entire stimulus sets, this framework enables similarity to be defined at the level of individual stimuli, revealing inputs that elicit convergent versus divergent representations across models. Using this instance-level notion of similarity, we identify systematic input properties that predict representational convergence versus divergence across vision and language model families. We also construct universal embedding spaces for brain representations across individuals and cortical regions, enabling instance-level comparison of representational agreement across stages of the human visual hierarchy. Finally, we apply the same barycentric alignment framework to purely unimodal vision and language models and find that post-hoc alignment into a shared space yields image text similarity scores that closely track human cross-modal judgments and approach the performance of contrastively trained vision-language models. This strikingly suggests that independently learned representations already share sufficient geometric structure for human-aligned cross-modal comparison. Together, these results show that resolving representational similarity at the level of individual stimuli reveals phenomena that cannot be detected by set-level comparison metrics.

Figures (18)

• Figure 1: Barycentric alignment into a universal embedding space. Representations from a diverse set of models, spanning architectures, training objectives, and scales, are aligned via a barycentric procedure into a shared embedding space by factoring out nuisance symmetries.
• Figure 1: Heatmap showing consistency of universal similarity scores across different model families (left), and across different architectures (right)
• Figure 2: Instance-level representational consistency is conserved across model pool constructions. Pairwise correlations of instance-level representational consistency scores computed from different model pools: (a) vision models and (b) language models, defined by different criteria (architectural family, scale, random mixtures).
• Figure 2: Universal Similarity scores depend strongly on the model weights used in the model pool, with substantial differences observed between models with random weights and those with trained weights.
• Figure 3: Representative images from the Animals parent category with the highest and lowest instance-level representational consistency scores, derived from a pooled ensemble of supervised models with varied architectures and scales.