Approximate Replicability in Learning

TL;DR

This work introduces three weakened notions of replicability for PAC learning—Pointwise Replicability, Approximate Replicability, and Semi-Replicability—aimed at bypassing the prohibitive costs of full replicability. It presents transformations showing any standard learner can be boosted to these notions with near-optimal sample costs for constant replication parameters: $O(d/\alpha^{2})$ for Pointwise and Approximate, and $O(d^{2}/(\alpha^{2}))$ labeled samples for Semi-Replicable learning, with shared randomness proven necessary. The paper also provides improved algorithms for one-dimensional thresholds under Approximate Replicability, tight lower bounds (including dependence on VC and Littlestone dimensions), and a systematic exploration of shared unlabeled data within Semi-Replicable learning. Overall, it formalizes a practical spectrum between distribution-dependent and distribution-free learning stability, showing how to trade replication strength for data efficiency and revealing fundamental limits dictated by randomness and complexity. The results have implications for robust deployment of learning systems where reproducibility across renewals and data shifts is critical, especially when public unlabeled data or shared randomness is available.

Abstract

Replicability, introduced by (Impagliazzo et al. STOC '22), is the notion that algorithms should remain stable under a resampling of their inputs (given access to shared randomness). While a strong and interesting notion of stability, the cost of replicability can be prohibitive: there is no replicable algorithm, for instance, for tasks as simple as threshold learning (Bun et al. STOC '23). Given such strong impossibility results we ask: under what approximate notions of replicability is learning possible? In this work, we propose three natural relaxations of replicability in the context of PAC learning: (1) Pointwise: the learner must be consistent on any fixed input, but not across all inputs simultaneously, (2) Approximate: the learner must output hypotheses that classify most of the distribution consistently, (3) Semi: the algorithm is fully replicable, but may additionally use shared unlabeled samples. In all three cases, for constant replicability parameters, we obtain sample-optimal agnostic PAC learners: (1) and (2) are achievable for ``free" using $Θ(d/α^2)$ samples, while (3) requires $Θ(d^2/α^2)$ labeled samples.

Figures (1)

• Figure 1: An example of sets $B_{y}$ covering $[-1, 1]^{2}$. The four regions must cover the whole square and correctness constraints require $B_{{\rm left}}$ (which consists of the blue $B_{(-1, 1)}$ and orange $B_{(-1, -1)}$ regions) to contain the full left border. Similar constraints hold on $B_{{\rm right}}, B_{{\rm up}}, B_{{\rm down}}$. Then, regardless of how the square is colored, at least one pair of opposing regions (either yellow and orange or blue and green) must intersect. In the given example, yellow $B_{(1, 1)}$ and orange $B_{(-1, -1)}$ intersect.

Theorems & Definitions (102)

• Definition 1.1: Pointwise Replicability
• Definition 1.2: Approximate Replicability
• Definition 1.3: Semi-Replicability
• Theorem 1.4: Informal \ref{['thm:predict-formal']}
• Theorem 1.5
• Theorem 1.6
• Theorem 1.7
• Proposition 1.7
• Theorem 1.8
• Theorem 1.9