The Triangle of Similarity: A Multi-Faceted Framework for Comparing Neural Network Representations

TL;DR

The paper tackles when different neural networks learn similar concepts by introducing the Triangle of Similarity, a holistic framework that combines static representational similarity ($CKA$ and Procrustes), functional similarity (Linear Mode Connectivity for same-architecture pairs and Predictive Similarity via Jensen–Shannon Divergence for different architectures), and sparsity-based robustness through pruning. It empirically analyzes CNNs, Vision Transformers, and Vision-Language Models on in-distribution and out-of-distribution data, revealing that architectural families form distinct representational clusters and that task accuracy is often more sensitive to pruning than the core representations. A key finding is the strong cross-view correlation between static similarity and sparsity robustness ($r=0.882$) with notable metric disagreements, underscoring the value of integrating multiple analytical lenses. The framework provides a practical toolkit for model comparison and selection in scientific contexts, enabling more robust interpretation of whether models converge on similar internal mechanisms across tasks and data regimes.

Abstract

Comparing neural network representations is essential for understanding and validating models in scientific applications. Existing methods, however, often provide a limited view. We propose the Triangle of Similarity, a framework that combines three complementary perspectives: static representational similarity (CKA/Procrustes), functional similarity (Linear Mode Connectivity or Predictive Similarity), and sparsity similarity (robustness under pruning). Analyzing a range of CNNs, Vision Transformers, and Vision-Language Models using both in-distribution (ImageNetV2) and out-of-distribution (CIFAR-10) testbeds, our initial findings suggest that: (1) architectural family is a primary determinant of representational similarity, forming distinct clusters; (2) CKA self-similarity and task accuracy are strongly correlated during pruning, though accuracy often degrades more sharply; and (3) for some model pairs, pruning appears to regularize representations, exposing a shared computational core. This framework offers a more holistic approach for assessing whether models have converged on similar internal mechanisms, providing a useful tool for model selection and analysis in scientific research.

Figures (5)

• Figure 1: The Triangle of Similarity Framework. Our approach integrates three complementary views to form a holistic comparison: (1) static similarity of internal representations, (2) functional similarity in weight or output space, and (3) sparsity similarity as models undergo stress via pruning.
• Figure 2: Global similarity landscape on OOD data. CKA (top) and Procrustes (bottom) heatmaps evaluated on CIFAR-10 images reveal that architectural clustering remains robust even when processing out-of-distribution data. High similarity is observed within CNNs and within Transformers, while cross-family similarity remains low.
• Figure 3: Effect of Pruning.Top: CKA self-similarity (pruned vs. original model) degrades as pruning increases. Bottom: Top-1 accuracy degrades more sharply than self-similarity, especially for Transformers like DeiT and ViT.
• Figure 4: Functional vs. Representational Divergence Under Pruning. The top panel shows the Linear Mode Connectivity (LMC) Barrier Height (Functional Divergence) between the original and pruned model ($M$ vs. $M^{(s)}$). The bottom panel shows the Average CKA Self-Similarity (Representational Divergence). The LMC Barrier rises more sharply and at lower sparsity levels than the CKA Self-Similarity drops, suggesting functional proximity is a more fragile property than representational structure.
• Figure 5: Cross-View Statistical Analysis.Top: A strong positive correlation (Pearson's $r=0.882$) exists between static similarity and robustness under sparsity. Bottom: CKA and Procrustes scores, while correlated, show several cases of high disagreement ($>0.15$), confirming that a multi-metric view is necessary.