Universality in Transfer Learning for Linear Models
Reza Ghane, Danil Akhtiamov, Babak Hassibi
TL;DR
The paper addresses transfer learning for overparameterized linear models trained with SGD, establishing that both regression and binary classification generalization/error outcomes are universal with respect to target distributions, depending only on first- and second-order statistics. It develops a rigorous Gaussian-universality framework that extends beyond Gaussian designs and applies it to both fine-tuning performance and implicit-regularization objectives, including mixtures of distributions. The main contribution is a pair of universality theorems showing that training outcomes with non-Gaussian data match those with matching Gaussian designs, enabling precise performance characterizations for regression and classification under pretraining and target-data fine-tuning. Practically, this yields exact transfer-learning conditions, scalar-covariance reductions, and regime-dependent insights into when fine-tuning improves or harms transfer, validated by numerical experiments across diverse distributions.
Abstract
We study the problem of transfer learning and fine-tuning in linear models for both regression and binary classification. In particular, we consider the use of stochastic gradient descent (SGD) on a linear model initialized with pretrained weights and using a small training data set from the target distribution. In the asymptotic regime of large models, we provide an exact and rigorous analysis and relate the generalization errors (in regression) and classification errors (in binary classification) for the pretrained and fine-tuned models. In particular, we give conditions under which the fine-tuned model outperforms the pretrained one. An important aspect of our work is that all the results are "universal", in the sense that they depend only on the first and second order statistics of the target distribution. They thus extend well beyond the standard Gaussian assumptions commonly made in the literature. Furthermore, our universality results extend beyond standard SGD training to the test error of a classification task trained using a ridge regression.