Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contextures: The Mechanism of Representation Learning

Runtian Zhai

TL;DR

This work proposes the contexture theory to mathematically characterize representation learning via the association between inputs and a context variable. By treating representations as the top singular functions of the joint X-A operator T_{P^+}, it explains why large models benefit only up to a point and why richer contexts are essential for further progress. The dissertation develops variational objectives (SVME, KISE) to extract the top-d eigenspace, introduces mixing operations (convolution, convex combination, concatenation) to create better contexts, and provides statistical generalization bounds through STKR and context complexity. It also analyzes intrinsic encoder evaluation, distribution shift, and practical tabular data improvements via context mixing, offering a unified lens on diverse pretraining paradigms. The results highlight that context design, rather than sheer scale, governs transferable performance and generalization, with the scalability of contexts offering a path to more robust foundation models.

Abstract

This dissertation establishes the contexture theory to mathematically characterize the mechanism of representation learning, or pretraining. Despite the remarkable empirical success of foundation models, it is not very clear what representations they learn, and why these representations are useful for various downstream tasks. A scientific understanding of representation learning is critical, especially at this point when scaling up the model size is producing diminishing returns, and designing new pretraining methods is imperative for further progress. Prior work treated different representation learning methods quite differently, whereas the contexture theory provides a unified framework for analyzing these methods. The central argument is that a representation is learned from the association between the input X and a context variable A. We prove that if an encoder captures the maximum information of this association, in which case we say that the encoder learns the contexture, then it will be optimal on the class of tasks that are compatible with the context. We also show that a context is the most useful when the association between X and A is neither too strong nor too weak. The important implication of the contexture theory is that increasing the model size alone will achieve diminishing returns, and further advancements require better contexts. We demonstrate that many pretraining objectives can learn the contexture, including supervised learning, self-supervised learning, generative models, etc. Then, we introduce two general objectives -- SVME and KISE, for learning the contexture. We also show how to mix multiple contexts together, an effortless way to create better contexts from existing ones. Then, we prove statistical learning bounds for representation learning. Finally, we discuss the effect of the data distribution shift from pretraining to the downstream task.

Contextures: The Mechanism of Representation Learning

TL;DR

This work proposes the contexture theory to mathematically characterize representation learning via the association between inputs and a context variable. By treating representations as the top singular functions of the joint X-A operator T_{P^+}, it explains why large models benefit only up to a point and why richer contexts are essential for further progress. The dissertation develops variational objectives (SVME, KISE) to extract the top-d eigenspace, introduces mixing operations (convolution, convex combination, concatenation) to create better contexts, and provides statistical generalization bounds through STKR and context complexity. It also analyzes intrinsic encoder evaluation, distribution shift, and practical tabular data improvements via context mixing, offering a unified lens on diverse pretraining paradigms. The results highlight that context design, rather than sheer scale, governs transferable performance and generalization, with the scalability of contexts offering a path to more robust foundation models.

Abstract

This dissertation establishes the contexture theory to mathematically characterize the mechanism of representation learning, or pretraining. Despite the remarkable empirical success of foundation models, it is not very clear what representations they learn, and why these representations are useful for various downstream tasks. A scientific understanding of representation learning is critical, especially at this point when scaling up the model size is producing diminishing returns, and designing new pretraining methods is imperative for further progress. Prior work treated different representation learning methods quite differently, whereas the contexture theory provides a unified framework for analyzing these methods. The central argument is that a representation is learned from the association between the input X and a context variable A. We prove that if an encoder captures the maximum information of this association, in which case we say that the encoder learns the contexture, then it will be optimal on the class of tasks that are compatible with the context. We also show that a context is the most useful when the association between X and A is neither too strong nor too weak. The important implication of the contexture theory is that increasing the model size alone will achieve diminishing returns, and further advancements require better contexts. We demonstrate that many pretraining objectives can learn the contexture, including supervised learning, self-supervised learning, generative models, etc. Then, we introduce two general objectives -- SVME and KISE, for learning the contexture. We also show how to mix multiple contexts together, an effortless way to create better contexts from existing ones. Then, we prove statistical learning bounds for representation learning. Finally, we discuss the effect of the data distribution shift from pretraining to the downstream task.
Paper Structure (125 sections, 68 theorems, 343 equations, 25 figures, 12 tables, 6 algorithms)

This paper contains 125 sections, 68 theorems, 343 equations, 25 figures, 12 tables, 6 algorithms.

Table of Contents

  1. Introduction to the Contexture Theory
  2. Central Theme: Representations from Association
  3. Takeaways.
  4. Chapter 2.
  5. Chapter 3.
  6. Chapter 4.
  7. Chapter 5.
  8. Chapter 6.
  9. Contexts: Definition and Examples
  10. Labels.
  11. Random transformations.
  12. Graphs.
  13. Feature maps and teacher models.
  14. Spectral Properties of a Context
  15. Three Types of Access and Example Contexts
  16. ...and 110 more sections

Key Result

Lemma 1.4

For all $i$, we have $\lambda_i = \kappa_i \in [0,1]$. And if $\lambda_i > 0$, then we have $\mu_i = \lambda_i^{-\frac{1}{2}} T_{P^+} \nu_i$, and $\nu_i = \lambda_i^{-\frac{1}{2}} T_{P^+}^{*} \mu_i$.

Figures (25)

  • Figure 1: Illustration of the modern ML paradigm driven by foundation models and representation learning, using language models as an example in the parentheses.
  • Figure 2: The association between $X$ and $A$ determines the shape of the spectrum.
  • Figure 3: Illustration of a transformation graph.
  • Figure 4: Two widely used self-supervised learning algorithms with random transformations. Left: Multi-view learning. Right: Reconstruction.
  • Figure 5: Convolution and convex combination of multiple transformations on images.
  • ...and 20 more figures

Theorems & Definitions (114)

  • Definition 1.1
  • Remark 1.2
  • Definition 1.3
  • Lemma 1.4: Duality property
  • Lemma 1.5: Spectral decomposition
  • Corollary 1.6
  • Definition 1.7
  • Definition 1.8
  • Remark 1.9
  • Definition 1.10
  • ...and 104 more