On the Computational Landscape of Replicable Learning
Alkis Kalavasis, Amin Karbasi, Grigoris Velegkas, Felix Zhou
TL;DR
This paper investigates the computational landscape of replicable learning, examining its connections to online learning, SQ learning, and differential privacy. It presents both negative results (computational separations, e.g., replicable PAC learnability vs. online learnability under one-way functions) and positive results (black-box lifting and transformations that extend replicable learning from uniform marginals to broader distributions). A central contribution is a decision-tree–complexity–dependent lifting framework that transfers replicability across marginals, enabling efficient replicable learning for parities under certain nonuniform distributions. Additionally, the authors show how to convert pure DP learners into replicable learners with runtimes that are polynomial in accuracy, confidence, and replicability but exponential in the hypothesis class’s representation dimension. Overall, the work advances understanding of when and how replicability can be achieved efficiently and delineates the tradeoffs with privacy and online learning, with practical implications for robust ML deployment under stability constraints.
Abstract
We study computational aspects of algorithmic replicability, a notion of stability introduced by Impagliazzo, Lei, Pitassi, and Sorrell [2022]. Motivated by a recent line of work that established strong statistical connections between replicability and other notions of learnability such as online learning, private learning, and SQ learning, we aim to understand better the computational connections between replicability and these learning paradigms. Our first result shows that there is a concept class that is efficiently replicably PAC learnable, but, under standard cryptographic assumptions, no efficient online learner exists for this class. Subsequently, we design an efficient replicable learner for PAC learning parities when the marginal distribution is far from uniform, making progress on a question posed by Impagliazzo et al. [2022]. To obtain this result, we design a replicable lifting framework inspired by Blanc, Lange, Malik, and Tan [2023] that transforms in a black-box manner efficient replicable PAC learners under the uniform marginal distribution over the Boolean hypercube to replicable PAC learners under any marginal distribution, with sample and time complexity that depends on a certain measure of the complexity of the distribution. Finally, we show that any pure DP learner can be transformed to a replicable one in time polynomial in the accuracy, confidence parameters and exponential in the representation dimension of the underlying hypothesis class.