A Random Matrix Theory of Masked Self-Supervised Regression

TL;DR

This work develops a high-dimensional random-matrix theory for masked self-supervised regression (SSR), where a matrix-valued predictor emerges from aggregating across masking patterns. By deriving a deterministic equivalent for the SSR matrix and sharp limits for generalization and training errors, it reveals how data geometry, via the population covariance $\Sigma$ and the aspect ratio $\alpha=n/d$, governs SSL generalization. The spectrum of the SSR predictor is characterized by a self-consistent resolvent, yielding BBP-type phase transitions in the spiked-covariance setting and universality results in the ridge-less, diagonal-covariance regime. Case studies on spiked covariance and AR(1) Toeplitz models show regimes where SSR outperforms PCA and delineate how temporal correlations shape the inductive bias of masked SSL. Overall, the paper provides principled, explicit connections between data structure, SSR optimization, and spectral behavior in the high-dimensional limit, with implications for understanding the advantages and limitations of SSL objectives over classical methods.

Abstract

In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor rather than a single vector-valued estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik--Ben Arous--Péché (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods

Figures (6)

• Figure 1: Generalization and Training Error for the SSR estimator for Gaussian data with a Toeplitz covariance $\Sigma_{i,j} = \rho^{|i-j|}$, for different values of $\rho$. The value $\rho=0$ corresponds to $\Sigma=I_d$. Solid lines correspond to the asymptotic limit, while dots correspond to the empirical error. In all curves, $d = 200$ and $\lambda = 10^{-4}$.
• Figure 2: Left: Spectrum of the SSR matrix $\hat{A}$ for Gaussian Data with a Toeplitz Covariance parametrized by $\rho \in (0,1)$, that is: $\Sigma_{i,j} = \rho^{|i-j|}$. The blue lines are the empirical spectrum, while the red lines are predicted by \ref{['thm:det_equivalent_A_hat']}. In both experiments $\lambda=0.01$, $d =1000$ and $\alpha = 3$. Right: Spectral density of $\hat{A}$ and the predicted spectral density from \ref{['corollary:universality']} for Gaussian data with covariance $\Sigma = C_{\beta}\mathrm{diag}(1, 2^{-\beta}, \dots, d^{-\beta})$, with $C_{\beta}$ such that $\mathrm{Tr}(\Sigma) = 1$. In all plots $\lambda = 10^{-5}$ and $d = 500$.
• Figure 3: Empirical spectral density of $\hat{A}$ for Gaussian, isotropic data, compared with the spectrum predicted by \ref{['thm:det_equivalent_A_hat']}. The dimension is $d=2000$ and $\lambda = 0.01$. On the left, $\alpha = 0.6$, and on the right, $\alpha = 1.5$.
• Figure 4: Left: Comparison of the SSR estimator and PCA for a spiked covariance $\Sigma = I_{d} + \theta vv^T$, for $v \sim \mathrm{Unif}(\mathbb{S}^{d-1})$. In the experiment, $d=300,\lambda = 0.01$ and PCA is applied for $p \in \{ 3, 10,30\}$. Right: Baik-Ben Arous-Peché transition for the SSR matrix $\hat{A}$ for Gaussian data with covariance $\Sigma = I_{d} + \theta vv^T$, for $v \sim \mathrm{Unif}(\mathbb{S}^{d-1})$ and varying $\theta$. The value of $\lambda = 10^{-5}$, $d = 2000$, and $\alpha = \frac{n}{d} = 2$.
• Figure 5: Left: Empirical generalization error for PCA and the self-supervised ridge estimator for Gaussian data with a Toeplitz covariance, for different values of $\rho$. In both pictures, $d=200$ and $\lambda=0.01$. Right: Generalization error for PCA and the self-supervised Ridge estimator for Gaussian data with a Toeplitz covariance, for different values of $\rho$. The sample size is fixed at $n=20.000$, the dimension is $d=300$ and $\lambda = 10^{-5}$. For PCA with $p$ directions, $\gamma = \frac{p}{d}$.

Theorems & Definitions (23)

• Remark 1
• Remark 2: Relationship with factored attention
• Remark 3
• Lemma 3.1
• Remark 4
• Lemma 3.2: Approximation error
• Remark 5
• Remark 6
• Theorem 1: Asymptotic Limit of the risk
• Remark 7