On the Generalization Error of Differentially Private Algorithms Via Typicality
Yanxiao Liu, Chun Hei Michael Shiu, Lele Wang, Deniz Gündüz
TL;DR
The paper addresses generalization guarantees for stochastic learning under privacy constraints by adopting an information-theoretic framework. It sharpens mutual-information bounds for differential privacy via stability and typicality arguments, providing explicit, computable bounds that improve prior work. It also introduces novel maximal-leakage bounds, including DP-specific results, which translate into tightened generalization guarantees. Together, these results yield tighter, practically usable tools for certifying generalization under privacy constraints and deepen the link between privacy stability and information leakage.
Abstract
We study the generalization error of stochastic learning algorithms from an information-theoretic perspective, with a particular emphasis on deriving sharper bounds for differentially private algorithms. It is well known that the generalization error of stochastic learning algorithms can be bounded in terms of mutual information and maximal leakage, yielding in-expectation and high-probability guarantees, respectively. In this work, we further upper bound mutual information and maximal leakage by explicit, easily computable formulas, using typicality-based arguments and exploiting the stability properties of private algorithms. In the first part of the paper, we strictly improve the mutual-information bounds by Rodríguez-Gálvez et al. (IEEE Trans. Inf. Theory, 2021). In the second part, we derive new upper bounds on the maximal leakage of learning algorithms. In both cases, the resulting bounds on information measures translate directly into generalization error guarantees.