Integrated representational signatures strengthen specificity in brains and models

TL;DR

SNF produces substantially sharper regional and model family-level separation than any single metric and yields robust composite similarity profiles, and clustering cortical regions using SNF-derived similarity scores reveals a clearer hierarchical organization that aligns closely with established anatomical and functional hierarchies of the visual cortex.

Abstract

The extent to which different neural or artificial neural networks (models) rely on equivalent representations to support similar tasks remains a central question in neuroscience and machine learning. Prior work has typically compared systems using a single representational similarity metric, yet each captures only one facet of representational structure. To address this, we leverage a suite of representational similarity metrics-each capturing a distinct facet of representational correspondence, such as geometry, unit-level tuning, or linear decodability-and assess brain region or model separability using multiple complementary measures. Metrics that preserve geometric or tuning structure (e.g., RSA, Soft Matching) yield stronger region-based discrimination, whereas more flexible mappings such as Linear Predictivity show weaker separation. These findings suggest that geometry and tuning encode brain-region- or model-family-specific signatures, while linearly decodable information tends to be more globally shared across regions or models. To integrate these complementary representational facets, we adapt Similarity Network Fusion (SNF), a framework originally developed for multi-omics data integration. SNF produces substantially sharper regional and model family-level separation than any single metric and yields robust composite similarity profiles. Moreover, clustering cortical regions using SNF-derived similarity scores reveals a clearer hierarchical organization that aligns closely with established anatomical and functional hierarchies of the visual cortex-surpassing the correspondence achieved by individual metrics.

Figures (3)

• Figure 1: A. Brain region separability under $d'$. Columns correspond to seven similarity metrics, including two fusion-based methods (SNF, average) and five commonly used representational metrics. The color bar is capped at 22. B. Mean separability score on NSD. Scores are shown in their native scales. C. Same analysis as in A, applied to vision model families.
• Figure 2: The heatmap shows the SNF-fused similarity matrix reordered by leaf ordering. Leaf labels are formatted as $region\_subject$ and colored by the cluster they belong to; dendrogram cuts yield up to ten flat clusters aligned with canonical categories. Considering high correlations across regions of the same subject caused by fMRI property, we zeroed out these similarity values to exclude the subject bias.
• Figure 3: Cross-regional relationships derived from five similarity measures using PCA analysis on NSD data. Each point represents a brain region instance, and text labels indicate centroid positions. SNF fusion shows best intra-class compactness and inter-class separation.