Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGD

TL;DR

This paper derives a more flatness-leveraging IT bound for the flatness-favoring SGD, which indicates the learned models generalize better if the large-variance directions of the final weight covariance have small local curvatures in the loss landscape.

Abstract

Information-theoretic (IT) generalization bounds have been used to study the generalization of learning algorithms. These bounds are intrinsically data- and algorithm-dependent so that one can exploit the properties of data and algorithm to derive tighter bounds. However, we observe that although the flatness bias is crucial for SGD's generalization, these bounds fail to capture the improved generalization under better flatness and are also numerically loose. This is caused by the inadequate leverage of SGD's flatness bias in existing IT bounds. This paper derives a more flatness-leveraging IT bound for the flatness-favoring SGD. The bound indicates the learned models generalize better if the large-variance directions of the final weight covariance have small local curvatures in the loss landscape. Experiments on deep neural networks show our bound not only correctly reflects the better generalization when flatness is improved, but is also numerically much tighter. This is achieved by a flexible technique called "omniscient trajectory". When applied to Gradient Descent's minimax excess risk on convex-Lipschitz-Bounded problems, it improves representative IT bounds' $Ω(1)$ rates to $O(1/\sqrt{n})$. It also implies a by-pass of memorization-generalization trade-offs.

Figures (17)

• Figure 1.1: wang_generalization_2021's bound for ResNet-18 on CIFAR-10 under varied flatness.
• Figure 3.1: The relationship between the original, omniscient, and SGLD-like trajectories, and illustrative examples of of alignment and misalignment between flatness and output weight covariance. The two trajectories have decoupled roles: the omniscient trajectory optimizes the whole bound while the SGLD-like trajectory bounds the MI for the omniscient trajectory.
• Figure 4.1: Thm. \ref{['theorem:flatness']} ($\lambda = 10^9$) on CIFAR-10 and ResNet-18 with varied learning rate and batch size.
• Figure 4.2: Numerical results for ResNet-18 on CIFAR-10 under varied training data usage, label noise level, width, depth, and weight decay. The isotropic version of \ref{['lemma:neu']} is used.
• Figure C.1: the isotropic version of \ref{['lemma:neu']} for ResNet-18 on CIFAR-10 under varied flatness.

Theorems & Definitions (37)

• Lemma 2.1: xu_information-theoretic_2017
• Definition 1: SGLD-Like Trajectory
• Proposition 1: Theorem 2 of wang_generalization_2021
• Proposition 2: Proposition 8 of neu_information-theoretic_2021
• Definition 2: Omniscient Trajectory
• Theorem 1
• Theorem 2
• Theorem 3
• Lemma A.1
• Corollary A.1