Information Theoretic Lower Bounds for Information Theoretic Upper Bounds
Roi Livni
TL;DR
This work shows that in stochastic convex optimization, achieving nontrivial generalization cannot be fully explained by information-theoretic upper bounds alone without incurring dimension-dependent information leakage. By constructing an information-theoretic lower bound via fingerprinting techniques, it demonstrates that any learning algorithm with nontrivial true risk must leak at least $\tilde{\Omega}\left( \dfrac{d}{\varepsilon^5 m^6(\varepsilon)}\right)$ bits of information about the sample, tying the information requirement directly to the ambient dimension. Consequently, dimension-independent, information-theoretic generalization bounds cannot capture the optimal rates of key algorithms like SGD and Regularized ERM in SCO, unless the sample size grows with dimension or other structural constraints are imposed. The results delineate the limitations of MI-based analyses and clarify when information leakage becomes a fundamental bottleneck for generalization in high-dimensional convex learning problems.
Abstract
We examine the relationship between the mutual information between the output model and the empirical sample and the generalization of the algorithm in the context of stochastic convex optimization. Despite increasing interest in information-theoretic generalization bounds, it is uncertain if these bounds can provide insight into the exceptional performance of various learning algorithms. Our study of stochastic convex optimization reveals that, for true risk minimization, dimension-dependent mutual information is necessary. This indicates that existing information-theoretic generalization bounds fall short in capturing the generalization capabilities of algorithms like SGD and regularized ERM, which have dimension-independent sample complexity.