Tighter CMI-Based Generalization Bounds via Stochastic Projection and Quantization
Milad Sefidgaran, Kimia Nadjahi, Abdellatif Zaidi
TL;DR
This work tackles tightening conditional mutual information (CMI) based generalization bounds by introducing stochastic projection and lossy compression into the CMI framework. By projecting the learned model to a lower-dimensional subspace using a random matrix $\Theta$ and then compressing via a lossy map to $\hat{W}$, the authors derive a disintegrated CMI bound that remains informative even in overparameterized regimes where classic CMI or MI bounds fail. The main result shows a bound on the generalization error that decays as $\mathcal{O}(\mathcal{LR}/\sqrt{n})$ in several SCO variants (and $\mathcal{O}(1/\sqrt{n})$ in the generalized linear settings), while also providing memorization-insensitive guarantees by constructing projection-based auxiliary algorithms with similar generalization performance but reduced memorization signatures. These results offer a constructive resolution to memorization questions and suggest design principles for learning systems where compressibility controls generalization and privacy concerns. The work thereby broadens the applicability of information-theoretic generalization analysis to more complex, high-dimensional learning scenarios.
Abstract
In this paper, we leverage stochastic projection and lossy compression to establish new conditional mutual information (CMI) bounds on the generalization error of statistical learning algorithms. It is shown that these bounds are generally tighter than the existing ones. In particular, we prove that for certain problem instances for which existing MI and CMI bounds were recently shown in Attias et al. [2024] and Livni [2023] to become vacuous or fail to describe the right generalization behavior, our bounds yield suitable generalization guarantees of the order of $\mathcal{O}(1/\sqrt{n})$, where $n$ is the size of the training dataset. Furthermore, we use our bounds to investigate the problem of data "memorization" raised in those works, and which asserts that there are learning problem instances for which any learning algorithm that has good prediction there exist distributions under which the algorithm must "memorize" a big fraction of the training dataset. We show that for every learning algorithm, there exists an auxiliary algorithm that does not memorize and which yields comparable generalization error for any data distribution. In part, this shows that memorization is not necessary for good generalization.