Bagging Provides Assumption-free Stability

TL;DR

This work answers the question of how stable bagging is when applied to an arbitrary base algorithm without distributional assumptions. It introduces average-case stability and proves a finite-sample guarantee for derandomized bagging (and variants) with bounded outputs, showing that stability holds whenever $\delta\varepsilon^2 \gtrsim \frac{1}{n}\cdot\frac{p}{1-p}$ (with refinements involving the resampling covariance $q$). The results extend to unbounded outputs via data-dependent scaling and adaptive clipping, establish tightness for subbagging, and demonstrate that worst-case stability cannot be guaranteed under the same conditions. Empirically, subbagging stabilizes highly unstable base learners across settings, supporting distribution-free uncertainty quantification and robust predictive intervals. Overall, the paper provides a principled, assumption-free stabilization mechanism for bagging with broad implications for generalization and inference.

Abstract

Bagging is an important technique for stabilizing machine learning models. In this paper, we derive a finite-sample guarantee on the stability of bagging for any model. Our result places no assumptions on the distribution of the data, on the properties of the base algorithm, or on the dimensionality of the covariates. Our guarantee applies to many variants of bagging and is optimal up to a constant. Empirical results validate our findings, showing that bagging successfully stabilizes even highly unstable base algorithms.

Figures (3)

• Figure 1: Distribution of leave-one-out perturbations for logistic regression (red) and subbagged logistic regression (blue), with $n=500$ and $d=200$. (See Section \ref{['sec-experiments']} for details on this simulation.)
• Figure 2: Phase diagram comparing \ref{['thm:upper', 'thm:lower']}, with $n=500, p=0.5$.
• Figure 3: Simulation results comparing the stability of subbagging $\widetilde{\mathcal{A}}_B$ to that of the corresponding base algorithm $\mathcal{A}$. Left: Histogram of leave-one-out perturbations. Right: for each $\varepsilon$, the smallest $\delta$ such that the algorithm is $\left(\varepsilon, \delta\right)$-stable in the sense of Definition \ref{['def:average-case-stability']}. Higher curves thus represent greater instability. In all settings, $m=n/2$ and $B = 10000$.

Theorems & Definitions (20)

• Definition 1: Stability---informal version
• Theorem 2: Main result---informal version
• Definition 3
• Definition 4
• Definition 6
• Theorem 8
• Theorem 9
• Theorem 10
• Definition 11
• Theorem 12