Asymptotics of resampling without replacement in robust and logistic regression
Pierre C. Bellec, Takuya Koriyama
TL;DR
The work analyzes bagging estimators built from subsamples drawn without replacement in the high-dimensional proportional regime ($n/p=\delta$) for robust linear regression and logistic regression. A key contribution is a simple nonlinear fixed-point equation for the limiting cross-estimator correlation, $\eta=F(\eta)$, allowing the limiting bagged-risk to be expressed as $\sigma^2/M + (1-1/M)\sigma^2\eta$ and enabling data-driven estimation of $\eta$ and $\sigma^2$. The authors prove existence and uniqueness of the fixed point, establish convergence of pairwise inner products to $\eta\sigma^2$, and provide estimators that consistently recover these limits from overlaps among subsamples. Numerical simulations with Huber/pseudo-Huber losses and logistic loss validate the theory and show how subsample size $q$ can nontrivially affect risk, including potential U-shaped risk curves in certain regimes. These results offer practical guidance for tuning subsample sizes in bagging under high dimensionality and contribute a rigorous fixed-point framework for resampling without replacement in robust and GLM contexts.
Abstract
This paper studies the asymptotics of resampling without replacement in the proportional regime where dimension $p$ and sample size $n$ are of the same order. For a given dataset $(X,y)\in \mathbb{R}^{n\times p}\times \mathbb{R}^n$ and fixed subsample ratio $q\in(0,1)$, the practitioner samples independently of $(X,y)$ iid subsets $I_1,...,I_M$ of $\{1,...,n\}$ of size $q n$ and trains estimators $\hatβ(I_1),...,\hatβ(I_M)$ on the corresponding subsets of rows of $(X, y)$. Understanding the performance of the bagged estimate $\barβ = \frac1M\sum_{m=1}^M \hatβ(I_1),...,\hatβ(I_M)$, for instance its squared error, requires us to understand correlations between two distinct $\hatβ(I_m)$ and $\hatβ(I_{m'})$ trained on different subsets $I_m$ and $I_{m'}$. In robust linear regression and logistic regression, we characterize the limit in probability of the correlation between two estimates trained on different subsets of the data. The limit is characterized as the unique solution of a simple nonlinear equation. We further provide data-driven estimators that are consistent for estimating this limit. These estimators of the limiting correlation allow us to estimate the squared error of the bagged estimate $\barβ$, and for instance perform parameter tuning to choose the optimal subsample ratio $q$. As a by-product of the proof argument, we obtain the limiting distribution of the bivariate pair $(x_i^T \hatβ(I_m), x_i^T \hatβ(I_{m'}))$ for observations $i\in I_m\cap I_{m'}$, i.e., for observations used to train both estimates.