Partial Soft-Matching Distance for Neural Representational Comparison with Partial Unit Correspondence

TL;DR

Partial soft-matching provides a principled and practical method for representational comparison under partial correspondence and achieves higher alignment precision across homologous brain areas than standard soft-matching, which is forced to match all units regardless of quality.

Abstract

Representational similarity metrics typically force all units to be matched, making them susceptible to noise and outliers common in neural representations. We extend the soft-matching distance to a partial optimal transport setting that allows some neurons to remain unmatched, yielding rotation-sensitive but robust correspondences. This partial soft-matching distance provides theoretical advantages -- relaxing strict mass conservation while maintaining interpretable transport costs -- and practical benefits through efficient neuron ranking in terms of cross-network alignment without costly iterative recomputation. In simulations, it preserves correct matches under outliers and reliably selects the correct model in noise-corrupted identification tasks. On fMRI data, it automatically excludes low-reliability voxels and produces voxel rankings by alignment quality that closely match computationally expensive brute-force approaches. It achieves higher alignment precision across homologous brain areas than standard soft-matching, which is forced to match all units regardless of quality. In deep networks, highly matched units exhibit similar maximally exciting images, while unmatched units show divergent patterns. This ability to partition by match quality enables focused analyses, e.g., testing whether networks have privileged axes even within their most aligned subpopulations. Overall, partial soft-matching provides a principled and practical method for representational comparison under partial correspondence.

Figures (12)

• Figure 1: Partial Soft-Matching Distance for Matching Tuning Curves.(A) Two toy networks $N_A$ and $N_B$; the layer of interest for alignment is shown in red. A partial matching recovers one-to-one correspondences between units with highly similar tuning curves. Line color encodes match strength (darker = stronger). By contrast, a purely soft-matching yields a spurious pair (hatched units, red dotted line). (B) The same metric can be used to rank voxel/unit tuning-curve similarity between two subjects’ responses $\{\bm{r}_1, \bm{r}_2\}$, when exposed to the same visual stimulus.
• Figure 2: Comparison of Balanced and Partial Soft-Matching.(a) We simulate two neural representations, $\bm{X}$ and $\bm{Y}$, with 120 and 190 neurons respectively. The first 100 neurons represent pure signal, while the rest are pure noise. Red denotes all pure signal neurons in the two representations. (b) The L-curve method selects the optimal mass regularization parameter $(=90/190\approx 0.47)$, successfully discarding noisy units.
• Figure 3: Model Selection Using Partial Soft-Matching. We simulate three synthetic representations to test whether partial soft-matching correctly identifies which of two candidate models---$\bm{Y}_a$ or $\bm{Y}_b$---shares more signal with a reference population $\bm{X}$ (100 units). $\bm{Y}_a$ contains all 100 signal units from $\bm{X}$ plus 60 noise units; $\bm{Y}_b$ contains 100 units, 80 of which match $\bm{X}$. The true fraction of shared units is known a priori, marked by a vertical gray line. With the L-curve-selected regularization, partial soft-matching yields correlation scores $s_{\mathrm{partial}}(\bm{X}, \bm{Y}_a)=0.715$ and $s_{\mathrm{partial}}(\bm{X}, \bm{Y}_b)=0.645$, correctly favoring $\bm{Y}_a$. Standard soft-matching fails, with $s_{\mathrm{sm}}(\bm{X}, \bm{Y}_a) = 0.339$ and $s_{\mathrm{sm}}(\bm{X}, \bm{Y}_b) = 0.415$, incorrectly preferring $\bm{Y}_b$ due to forced matching of noise.
• Figure 4: Aligning Voxel Responses Between Different Subjects in NSD. For each area, we plot the (i) partial soft-matching score at different mass regularization values and (ii) the mean noise ceilings of the voxels that were kept at that regularization. The alignment criterion consistently identifies low noise-ceiling voxels for exclusion.
• Figure 5: Evaluating Methods for Identifying (Un)matched Neurons in Deep Networks. We compare three methods for ranking convolutional kernels by alignment between two ResNet-$18$ models trained from different random initializations on ImageNet, across early, middle, and late layers. Removing low-alignment units identified by partial soft-matching yields alignment scores nearly identical to those obtained by removing kernels ranked least important via brute-force ablations, while correlation-based rankings perform poorly.