Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Bounds
Xinping Yi, Gaojie Jin, Xiaowei Huang, Shi Jin
TL;DR
The paper confronts the challenge of explaining generalization in deep networks by proposing a unified PAC-Bayesian framework that uses anisotropic Gaussian posteriors orchestrated by a structure-aware sensitivity matrix. By reframing the bound as a stochastic optimization over covariance and perturbation structure, the authors derive a family of bounds that recover existing results as special cases and tighten them through architecture-aware designs. Key contributions include a closed-form KL minimization with per-layer block-diagonal simplifications and multiple sensitivity shapes (diagonal, low-rank, circulant, Toeplitz) that encode architectural priors such as depth, weight sharing, and locality. The framework yields sharper, more interpretable bounds for CNNs, residual networks, and related architectures, offering a principled path toward geometry- and structure-aware generalization analysis in deep learning. Overall, the work provides a flexible, principled approach to bounding generalization by aligning posterior geometry with the loss landscape and network structure, with potential extensions to adversarial settings and graph-based models.
Abstract
Understanding the generalization behavior of deep neural networks remains a fundamental challenge in modern statistical learning theory. Among existing approaches, PAC-Bayesian norm-based bounds have demonstrated particular promise due to their data-dependent nature and their ability to capture algorithmic and geometric properties of learned models. However, most existing results rely on isotropic Gaussian posteriors, heavy use of spectral-norm concentration for weight perturbations, and largely architecture-agnostic analyses, which together limit both the tightness and practical relevance of the resulting bounds. To address these limitations, in this work, we propose a unified framework for PAC-Bayesian norm-based generalization by reformulating the derivation of generalization bounds as a stochastic optimization problem over anisotropic Gaussian posteriors. The key to our approach is a sensitivity matrix that quantifies the network outputs with respect to structured weight perturbations, enabling the explicit incorporation of heterogeneous parameter sensitivities and architectural structures. By imposing different structural assumptions on this sensitivity matrix, we derive a family of generalization bounds that recover several existing PAC-Bayesian results as special cases, while yielding bounds that are comparable to or tighter than state-of-the-art approaches. Such a unified framework provides a principled and flexible way for geometry-/structure-aware and interpretable generalization analysis in deep learning.