Bridging Functional and Representational Similarity via Usable Information

TL;DR

A unified framework for quantifying the similarity between representations through the lens of usable information is presented, offering a rigorous theoretical and empirical synthesis across three key dimensions, establishing representational similarity as the limit of maximum granularity: input reconstruction.

Abstract

We present a unified framework for quantifying the similarity between representations through the lens of \textit{usable information}, offering a rigorous theoretical and empirical synthesis across three key dimensions. First, addressing functional similarity, we establish a formal link between stitching performance and conditional mutual information. We further reveal that stitching is inherently asymmetric, demonstrating that robust functional comparison necessitates a bidirectional analysis rather than a unidirectional mapping. Second, concerning representational similarity, we prove that reconstruction-based metrics and standard tools (e.g., CKA, RSA) act as estimators of usable information under specific constraints. Crucially, we show that similarity is relative to the capacity of the predictive family: representations that appear distinct to a rigid observer may be identical to a more expressive one. Third, we demonstrate that representational similarity is sufficient but not necessary for functional similarity. We unify these concepts through a task-granularity hierarchy: similarity on a complex task guarantees similarity on any coarser derivative, establishing representational similarity as the limit of maximum granularity: input reconstruction.

Figures (4)

• Figure 1: Flowchart summarizing the results of Section \ref{['sec:how']}. The inputs are highlighted with cyan contours. Arrows are color-coded as follows: blue for Corollary \ref{['cor:func_sim_v_stitchability']}, green for Equation \ref{['eq:rep_sim_mse']}, brown for Proposition \ref{['prop:monotonicity']} ($\mathcal{V} \subseteq \mathcal{W}$), amber for Proposition \ref{['prop:coarse_func_sim_under_v']} ($Y' = g(Y)$), and coral for Corollary \ref{['cor:representational_functional_under_v']}.
• Figure 2: Experimental Validation. (a) The wide distribution of accuracy differences confirms that stitching is often asymmetric. (b) The similarity hierarchy (Affine $>$ Ortho+Scale $>$ Ortho) confirms that representational similarity increases as the predictive family $\mathcal{V}$ becomes more expressive. (c) Coarse-task similarity (green) consistently upper-bounds fine-task similarity (blue), validating that usable information is strictly hierarchical with respect to task granularity.
• Figure 3: Metric Alignment. Comparison between standard similarity metrics (y-axis) and the representational similarity derived from an Orthogonal + Scaling stitcher (x-axis). The strong correlations (especially for CKA and RSA) support the hypothesis that these metrics effectively measure usable information under specific transformation groups.
• Figure 4: Rep. Similarity Implies Func. Similarity. Conditional probability of high functional similarity (y-axis) given representational similarity (x-axis). While high representational similarity guarantees functional equivalence (validating Corollary \ref{['cor:representational_functional_under_v']}), the reverse does not hold: high functional similarity frequently occurs despite low representational similarity.

Theorems & Definitions (27)

• Definition 3.1: Representation
• Definition 3.2: Markov blanket
• Definition 3.3: Functional Similarity
• Definition 3.4: Perfect Stitchability
• Proposition 3.5: Markov Blanket–Stitchability Equivalence under Optimality
• Corollary 3.6: Functional Similarity–Stitchability Equivalence
• Definition 3.7: Representational Similarity
• Proposition 3.8: Granular Similarity $\Rightarrow$ Coarser Similarity
• Remark 3.9: Representational as Special Case of Functional Similarity
• Corollary 3.10: Representational $\Rightarrow$ Functional Similarity