Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a Learning Theory of Representation Alignment

Francesco Insulla, Shuo Huang, Lorenzo Rosasco

TL;DR

This work develops a learning-theoretic framework for representation alignment across uni- and multi-modal models. It unifies kernel alignment (KA) with distance-based and independence-testing notions, including HSIC, MMD, and measure-based divergences, through spectral interpretations in RKHS. The paper formalizes stitching as a task-aware probe and shows that stitching error can be bounded by kernel alignment, connecting practical transfer-like operations to fundamental alignment metrics. It also provides an array of theoretical tools, such as KARE and spectral-task alignments, to quantify generalization and alignment trade-offs across modalities. Overall, the results position representation alignment as a principled learning-theoretic problem with concrete bounds and interpretable connections between diverse alignment notions.

Abstract

It has recently been argued that AI models' representations are becoming aligned as their scale and performance increase. Empirical analyses have been designed to support this idea and conjecture the possible alignment of different representations toward a shared statistical model of reality. In this paper, we propose a learning-theoretic perspective to representation alignment. First, we review and connect different notions of alignment based on metric, probabilistic, and spectral ideas. Then, we focus on stitching, a particular approach to understanding the interplay between different representations in the context of a task. Our main contribution here is relating properties of stitching to the kernel alignment of the underlying representation. Our results can be seen as a first step toward casting representation alignment as a learning-theoretic problem.

Towards a Learning Theory of Representation Alignment

TL;DR

This work develops a learning-theoretic framework for representation alignment across uni- and multi-modal models. It unifies kernel alignment (KA) with distance-based and independence-testing notions, including HSIC, MMD, and measure-based divergences, through spectral interpretations in RKHS. The paper formalizes stitching as a task-aware probe and shows that stitching error can be bounded by kernel alignment, connecting practical transfer-like operations to fundamental alignment metrics. It also provides an array of theoretical tools, such as KARE and spectral-task alignments, to quantify generalization and alignment trade-offs across modalities. Overall, the results position representation alignment as a principled learning-theoretic problem with concrete bounds and interpretable connections between diverse alignment notions.

Abstract

It has recently been argued that AI models' representations are becoming aligned as their scale and performance increase. Empirical analyses have been designed to support this idea and conjecture the possible alignment of different representations toward a shared statistical model of reality. In this paper, we propose a learning-theoretic perspective to representation alignment. First, we review and connect different notions of alignment based on metric, probabilistic, and spectral ideas. Then, we focus on stitching, a particular approach to understanding the interplay between different representations in the context of a task. Our main contribution here is relating properties of stitching to the kernel alignment of the underlying representation. Our results can be seen as a first step toward casting representation alignment as a learning-theoretic problem.

Paper Structure

This paper contains 40 sections, 7 theorems, 54 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Structure of the paper
  3. Preliminaries
  4. Setup
  5. Reality
  6. Uni/Multi-modal
  7. Representation alignment
  8. Frameworks for Representation Alignment
  9. Kernel alignment (KA)
  10. Empirical KA
  11. Representation interpretation of KA
  12. RKHS operator interpretation of KA
  13. Population version of KA
  14. Spectral Interpretation of KA
  15. Statistical properties of KA.
  16. ...and 25 more sections

Key Result

Lemma 1

Let $K_1, K_2$ be two kernels for different representations and $\widehat{K}_{1,n}, \widehat{K}_{2,n} \in \mathbb{R}^{n \times n}$ be kernel matrices generated by $n$ samples, then with probability at least $1-\delta$, we have

Figures (1)

  • Figure 1: Diagram illustrating the process of multi-modal learning. It contains spaces and measures of reality, objects/data, representation, and outputs as well as the functions connecting them. A detailed explanation of these symbols is in Section \ref{['sec:settings']}.

Theorems & Definitions (17)

  • Lemma 1
  • Remark 1: Other notions from independence testing
  • Lemma 2
  • Remark 2
  • Theorem 1
  • proof
  • Remark 3
  • Remark 4
  • Remark 5
  • Lemma 3
  • ...and 7 more