Is Algorithmic Stability Testable? A Unified Framework under Computational Constraints

TL;DR

This work provides a unified framework for the hardness of testing algorithmic stability of black-box learners under computational and data-constrained settings. It proves universal upper bounds on the power of any valid stability test, showing that limited budget, finite data spaces, or restricted sampling fundamentally cap detectability of stability via black-box procedures. The results generalize prior work by incorporating finite data spaces and computational budgets, and reveal that, in many practical regimes, testing stability is nearly as hard as exhaustive search. The findings motivate relaxing stability notions or partially knowing the algorithm as practical paths to feasible validation in real-world settings.

Abstract

Algorithmic stability is a central notion in learning theory that quantifies the sensitivity of an algorithm to small changes in the training data. If a learning algorithm satisfies certain stability properties, this leads to many important downstream implications, such as generalization, robustness, and reliable predictive inference. Verifying that stability holds for a particular algorithm is therefore an important and practical question. However, recent results establish that testing the stability of a black-box algorithm is impossible, given limited data from an unknown distribution, in settings where the data lies in an uncountably infinite space (such as real-valued data). In this work, we extend this question to examine a far broader range of settings, where the data may lie in any space -- for example, categorical data. We develop a unified framework for quantifying the hardness of testing algorithmic stability, which establishes that across all settings, if the available data is limited then exhaustive search is essentially the only universally valid mechanism for certifying algorithmic stability. Since in practice, any test of stability would naturally be subject to computational constraints, exhaustive search is impossible and so this implies fundamental limits on our ability to test the stability property for a black-box algorithm.

Figures (2)

• Figure 1: Black-box test with computational constraints. Here in round $r$, the generating process of the new data set $\mathcal{D}_{\ell}^{(r)}$ and random seed $\xi^{(r)}$ for training can depend on the input data and on all the past information from previous rounds---e.g., we may create a new data set by resampling from past data. The input random seeds $\zeta,\zeta_1,\zeta_2,\dots\stackrel{\textnormal{iid}}{\sim}\textnormal{Unif}[0,1]$ may be used to provide randomization if desired. The procedure stops at a data-dependent stopping time ${\widehat{r}}$, with the constraint that the total number of training points used in calls to $\mathcal{A}$ cannot exceed the budget $B_{\textnormal{train}}$.
• Figure 2: Black-box test with black-box models under computational constraints. Here in round $r$, the generating process of the new labeled data set $\mathcal{D}_{\ell}^{(r)}$ for training, the unlabeled data set $\mathcal{X}^{(r)}$ for evaluation and the new random seed $\xi^{(r)}$ can depend on the input data and on all the past information from previous rounds---e.g., we may create a new labeled data set and an unlabeled data set by resampling from past data. The input random seeds $\zeta,\zeta_1,\zeta_2,\dots\stackrel{\textnormal{iid}}{\sim}\textnormal{Unif}[0,1]$ may be used to provide randomization if desired. The procedure stops at a data-dependent stopping time ${\widehat{r}}$ (i.e., we may use any stopping criterion), subject to the computational constraints that (1) the total number of training points used in calls to $\mathcal{A}$ cannot exceed the budget $B_{\textnormal{train}}$ and (2) the total number of data points used to evaluate the fitted models cannot exceed the budget $B_{\textnormal{eval}}$.

Theorems & Definitions (20)

• Definition 1: Algorithmic stability
• Definition 2: Black-box test under computational constraints
• Theorem 1
• Theorem 2
• Proposition 1
• Lemma 1
• Lemma 2
• Lemma 3
• Definition 3: Black-box test with black-box models, under computational constraints
• Theorem 3