Asymptotic Theory of Iterated Empirical Risk Minimization, with Applications to Active Learning

TL;DR

This work analyzes two successive ERMs trained on the same dataset, where the first-stage predictions feed into the loss of the second-stage, and develops a sharp asymptotic theory in the high-dimensional regime $n/d=\Theta(1)$ for Gaussian-mixture data. The authors derive a finite collection of scalar order parameters that characterize the second-stage estimator and its test performance, with a key correction term $\chi$ capturing correlations induced by data reuse via a nested leave-one-out approach. They instantiate the theory in an active-learning setting, removing oracle and sample-splitting assumptions, and uncover a budget-allocation tradeoff and a selection-driven double-descent phenomenon, supported by real-data validation on pneumonia diagnosis. The results offer principled guidance for budget-constrained data acquisition and provide a rigorous framework to analyze data-reuse effects in iterated optimization procedures.

Abstract

We study a class of iterated empirical risk minimization (ERM) procedures in which two successive ERMs are performed on the same dataset, and the predictions of the first estimator enter as an argument in the loss function of the second. This setting, which arises naturally in active learning and reweighting schemes, introduces intricate statistical dependencies across samples and fundamentally distinguishes the problem from classical single-stage ERM analyses. For linear models trained with a broad class of convex losses on Gaussian mixture data, we derive a sharp asymptotic characterization of the test error in the high-dimensional regime where the sample size and ambient dimension scale proportionally. Our results provide explicit, fully asymptotic predictions for the performance of the second-stage estimator despite the reuse of data and the presence of prediction-dependent losses. We apply this theory to revisit a well-studied pool-based active learning problem, removing oracle and sample-splitting assumptions made in prior work. We uncover a fundamental tradeoff in how the labeling budget should be allocated across stages, and demonstrate a double-descent behavior of the test error driven purely by data selection, rather than model size or sample count.

Figures (6)

• Figure 1: Test error $\egen$ of the classifier $\hat{w}$\ref{['eq:main_rew_ERM']} trained on the subset $\mathcal{S}$, selected using a base classifier $\hat{w}_0$\ref{['eq:main_base_ERM']} trained on a fraction $\psi\in (0,\gamma)$ of the $n$ samples for $\alpha=8, \lambda=0.01$, losses $\ell(y,z),\ell_0(y,z)=1/2(y-z)^2$, and policy $\pi(u)=\mathds{1}_{|u|<\kappa^-(\gamma,\psi)}$. Different curves correspond to different total budgets $\gamma\in(0,1)$. Solid lines: theoretical predictions. Dots: numerical experiments in dimension $d=2000$; error bars represent one standard deviation over $20$ trials. Dashed line: test error of the base classifier. Inset: for $\gamma=0.7$, the dashed line represents the test error when only the newly selected subset$\mathcal{S}\setminus \mathcal{S}_0$ is used for the second ERM, with the base training set $\mathcal{S}_0$ being held-out.
• Figure 2: Test error as a function of the fraction $\psi\in(0,\gamma)$ allocated to the first stage of ERM, for the pneumonia diagnosis task on the chest X-ray dataset kermany2018identifying, pre-processed through a scattering transform with scale $4$ and $6$ angles andreux2020kymatio. The logistic loss $\ell(z,y)=\ell_0(z,y)=\log(1+\exp(-yz))$ was employed, with $\alpha=5.41, \lambda=0.01$. The samples closest to the decision boundary of the first classifier were retained, until the budget was saturated. Different curves represent different total budget $\gamma$. Shades represent one standard deviation over $100$ trials.
• Figure 3: Same setting as Fig. \ref{['fig:tradeoff']}, with $\alpha=1$. Dashed curves represent the selection policy $\pi(u)=\mathds{1}_{|u|>\kappa^+(\gamma,\psi)}$, where $\kappa^+(\gamma,\psi)=\sqrt{2q_0}{\rm erf}^{-1}\left(1-\gamma/1-\psi\right)$, corresponding to selecting the samples with higher margin. Solid lines still represent the small-margin selection policy $\pi(u)=\mathds{1}_{|u|<\kappa^-(\gamma,\psi)}$. The inset represents the case where oracle access of $\beta$ is available, so that no budget $\psi=0$ needs to be used to train the base classifier, and the data can be selected directly using the true margins as $\pi(\inprod{\beta,x_i})$.
• Figure 4: Test error $\egen$ achieved by the second classifier, with $\alpha=8,\gamma=0.3,\psi=0.1, \lambda_0=\lambda=0.001$, and square loss. A selection policy $\pi(u)=\mathds{1}_{|u|<\kappa^-}+\mathds{1}_{|u|>\kappa^+}$ was used, for $\kappa^-\in(0,\kappa^-(\gamma,\psi))$, and $\kappa^+$ determined from $\kappa^-$ by requiring that the budget constraint should be saturated. Solid lines: characterization of Theorem \ref{['theorem:main_theorem']}. Dots: numerical experiments in dimension $d=3000$. Error bars represent one standard deviation over $50$ trials.
• Figure 5: Test error , as a function of the normalized number of samples $\alpha=n/d$, of a logistic classifier ($\Tilde{\ell}(y,z)=\log(1+e^{-yz}),\lambda=0.01$) trained on a subset $\mathcal{S}\subset [n]$ pruned using a large-margin selection policy $\pi(u)=\mathds{1}_{|u|>\kappa^+}$. The base estimator was trained on the whole dataset ($\mathcal{S}_0=[n]$), also with $\Tilde{\ell}(y,z)=\log(1+e^{-yz}),\lambda=0.01$. The data is a binary isotropic Gaussian mixture with centroid norm $\norm{\mu}=0.8$. Different curves represent different thresholds $\kappa^+$, with higher values implying smaller training subsets $|\mathcal{S}|$. Solid lines: theoretical predictions of Theorem \ref{['theorem:main_theorem']}. Dots: numerical experiments in dimension $d=2000$. Error bars represent one standard deviation over $30$ trials.

Theorems & Definitions (71)

• Theorem 2.1
• Remark A.1: Properties of $O_{L_k}$
• Proposition B.1
• Lemma B.2
• proof
• Remark B.3
• Lemma B.4
• proof
• Lemma B.5
• proof