The Role of Randomness in Stability
Max Hopkins, Shay Moran
TL;DR
This work characterizes how much randomness is necessary to achieve stability in learning, tying replication and DP to a task's intrinsic stability via a weak-to-strong boosting principle. It shows that the randomness complexity of replicability and DP is controlled by the best deterministic replication probability (global stability) and that PAC learning with finite Littlestone dimension has bounded randomness cost, scaling with the excess error. The results provide tight relationships between global stability, certificate complexity, and DP complexity, and they resolve an open question by establishing an error-dependent bound in agnostic PAC learning governed by the Littlestone dimension. These insights yield practical guidance for designing randomness-efficient, stable learning algorithms and clarifying the limits of DP and replicability in complex tasks.
Abstract
Stability is a central property in learning and statistics promising the output of an algorithm $A$ does not change substantially when applied to similar datasets $S$ and $S'$. It is an elementary fact that any sufficiently stable algorithm (e.g.\ one returning the same result with high probability, satisfying privacy guarantees, etc.) must be randomized. This raises a natural question: can we quantify how much randomness is needed for algorithmic stability? We study the randomness complexity of two influential notions of stability in learning: replicability, which promises $A$ usually outputs the same result when run over samples from the same distribution (and shared random coins), and differential privacy, which promises the output distribution of $A$ remains similar under neighboring datasets. The randomness complexity of these notions was studied recently in (Dixon et al. ICML 2024) and (Cannone et al. ITCS 2024) for basic $d$-dimensional tasks (e.g. estimating the bias of $d$ coins), but little is known about the measures more generally or in complex settings like classification. Toward this end, we prove a `weak-to-strong' boosting theorem for stability: the randomness complexity of a task $M$ (either under replicability or DP) is tightly controlled by the best replication probability of any deterministic algorithm solving the task, a weak measure called `global stability' that is universally capped at $\frac{1}{2}$ (Chase et al. FOCS 2023). Using this, we characterize the randomness complexity of PAC Learning: a class has bounded randomness complexity iff it has finite Littlestone dimension, and moreover scales at worst logarithmically in the excess error of the learner. This resolves a question of (Chase et al. STOC 2024) who asked for such a characterization in the equivalent language of (error-dependent) `list-replicability'.