A Function-Space Stability Boundary for Generalization in Interpolating Learning Systems

TL;DR

The paper addresses when algorithmic stability explains generalization in modern interpolating learners by modeling training as a function-space trajectory and measuring single-sample perturbations along it. It introduces a contractivity-based stability certificate, obtained by unrolling a one-step discrepancy recursion, and proves a sufficiency bound linking the certificate to the generalization gap while also constructing a necessity regime showing stability cannot be a universal explanation. The framework is optimizer-agnostic and computable, enabling diagnostics from training primitives via estimators of the contractivity factors. Empirical validation on overparameterized linear regression demonstrates that certificate growth tracks generalization differences across optimizers, step sizes, and data geometry, thereby identifying regimes where stability explains generalization and regimes where geometry or implicit bias dominate. Overall, the work establishes a precise, transferable boundary for stability-based explanations and provides practical diagnostics for when to trust or doubt stability as the driver of generalization.

Abstract

Modern learning systems often interpolate training data while still generalizing well, yet it remains unclear when algorithmic stability explains this behavior. We model training as a function-space trajectory and measure sensitivity to single-sample perturbations along this trajectory. We propose a contractive propagation condition and a stability certificate obtained by unrolling the resulting recursion. A small certificate implies stability-based generalization, while we also prove that there exist interpolating regimes with small risk where such contractive sensitivity cannot hold, showing that stability is not a universal explanation. Experiments confirm that certificate growth predicts generalization differences across optimizers, step sizes, and dataset perturbations. The framework therefore identifies regimes where stability explains generalization and where alternative mechanisms must account for success.

Figures (5)

• Figure 1: Empirical relationship between the stability certificate and generalization error across all logged runs. The association is descriptive and is used to validate that the certificate behaves as a stability diagnostic under the controlled interventions of this section.
• Figure 2: Neighbor-selection ablation (GD, $\eta=0.2$). Curves show the certificate prefix and the probe discrepancy under random index replacement versus high-leverage index replacement, averaged over seeds.
• Figure 3: Optimizer comparison at the logged step sizes. Curves show certificate prefix and test MSE averaged over seeds. The purpose is diagnostic. Hyperparameters are not tuned to equalize convergence speed or interpolation across optimizers.
• Figure 4: Label-permutation intervention (GD, $\eta=0.2$). Curves show certificate prefix and test MSE averaged over seeds under clean labels versus permuted labels.
• Figure 5: Step-size sweep (GD). Larger step sizes increase certificate growth. Terminal test MSE changes only mildly at the reported scale, but the certificate responds monotonically to the stability-relevant control parameter.

Theorems & Definitions (24)

• Remark 3.1: Regularity for high-probability bounds
• Definition 3.2: Trajectory contractivity profile
• Lemma 4.1: Unrolling
• Remark 4.2: Worst-case nature
• Theorem 5.1: Trajectory contractivity implies stability-based generalization
• proof : Proof Sketch
• Theorem 5.2: Existence of non-stability generalization regimes
• proof : Proof sketch
• Proposition 6.1: One-step smoothness bound
• Lemma A.1: Measurability of $\Delta_t$ and $\mathsf{Cert}_T$