Contextures: Representations from Contexts

TL;DR

The contexture theory is established and it is proved that representation learning within various learning paradigms -- supervised, self-supervised, and manifold learning -- can all be studied from such a perspective.

Abstract

Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture. We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms -- supervised, self-supervised, and manifold learning -- can all be studied from such a perspective. We also prove that the representations that learn the contexture are optimal on those tasks that are compatible with the context. One important implication of the contexture theory is that once the model is large enough to approximate the top singular functions, further scaling up the model size yields diminishing returns. Therefore, scaling is not all we need, and further improvement requires better contexts. To this end, we study how to evaluate the usefulness of a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets.

Figures (11)

• Figure 1: Alignment between the learned representation and the top-$d$ eigenfunctions of $T_{{k_{X}^+}}$ on the abalone dataset. Solid curves: CCA. Dashed curves: mutual KNN. Depth here means the number of hidden layers.
• Figure 2: Association level vs, prediction error $\textnormal{err}_{d^*} = \min_d \textnormal{err}_d$ when $\Phi$ is d-dimensional, and the decay rate $\lambda$ for the RBF and KNN contexts on credit-approval and breast-w datasets. A larger $-\gamma$ (or $K/N$) indicates a lower association level, while a smaller $-\gamma$ (or $K/N$) corresponds to a higher association level. Across all four figures, we observe a U-shaped trend: prediction error increases at both extremes of low and high association. The estimated decay rate $\lambda$ consistently captures this behavior, serving as a proxy for the level of association.
• Figure 3: Metric illustration on abalone. Top row: context spectra. Bottom row: black solid curves are $\tau_d$ divided by $6$; red dashed curves are the actual downstream prediction error. We divide $\tau_d$ by $6$ to fit it in the same plot.
• Figure 4: Metric illustration on MNIST, similar to \ref{['fig:taud']}.
• Figure 5: Comparison of the downstream task between abalone and MNIST. The $y$-axis is the cosine similarity between the downstream task and the $i$-th eigenfunction.

Theorems & Definitions (42)

• Definition 1
• Definition 2
• Lemma 3: Duality property, zhai2023understanding, Proposition 1
• Lemma 4
• proof
• Definition 5
• Theorem 6: Proof in \ref{['app:proof-thm-supervised-cls']}
• Theorem 7: Proof in \ref{['app:proof-thm-supervised-bal-cls']}
• Theorem 8: Proof in \ref{['app:proof-thm-obj-regression']}
• Remark