Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regime

TL;DR

The paper tackles why overparameterized models generalize despite interpolating training data by developing data-dependent, high-probability bounds for the test error of the Gibbs posterior and posterior mean across all temperatures, using an integral representation of the log-partition function and PAC-Bayesian ideas. It demonstrates that these bounds are stable under Langevin Monte Carlo approximations and validates the approach on MNIST and CIFAR-10, showing nontrivial bounds for true labels and correct upper bounds for random labels. Through calibration and ergodic-mean approximations, the method yields tight, temperature-aware guarantees in the interpolation regime, linking high-temperature training behavior to low-temperature generalization signals. The work provides a practical framework for certifying LMC-based learners and offers insight into how generalization can emerge even when training errors are small on data designed to fail test performance.

Abstract

The paper provides data-dependent bounds on the test error of the Gibbs algorithm in the overparameterized interpolation regime, where low training errors are also obtained for impossible data, such as random labels in classification. The bounds are stable under approximation with Langevin Monte Carlo algorithms. Experiments on the MNIST and CIFAR-10 datasets verify that the bounds yield nontrivial predictions on true labeled data and correctly upper bound the test error for random labels. Our method indicates that generalization in the low-temperature, interpolation regime is already signaled by small training errors in the more classical high temperature regime.

Figures (7)

• Figure 1: SGLD on MNIST and CIFAR-10 with 8000 training examples, MNIST above and CIFAR-10 below, random labels on the left, correct labels on the right. Both random and true labels are trained with the same algorithm and parameters on a fully connected ReLU network with two hidden layers of 1000 and 1500 units, respectively. The calibration factor for MNIST is 0.19, for CIFAR-10 0.22. Train error, test error and our bound for the Gibbs posterior average of the 0-1 loss are plotted against $\beta$.
• Figure 2: A more detailed version of Figure \ref{['MC8k']} to illustrate how the bounds are computed.
• Figure 3: SGLD on MNIST and CIFAR-10 with 2000 training examples using BBCE loss function. The first row corresponds to MNIST and the second row to CIFAR-10. Random labels are shown on the left, correct labels on the right. Both random and true labels are trained with exactly the same algorithm and parameters on a fully connected ReLU network with two hidden layers of 1000 (respectively 1500) units. The calibration factor for MNIST is 0.19, for CIFAR-10 0.22. Train error, test error and our bound for a single-draw of the 0-1 loss are plotted against $\beta$.
• Figure 4: SGLD on MNIST and CIFAR-10 with 8000 training examples using BBCE loss function. The first two rows correspond to MNIST, and the remaining rows to CIFAR-10. Random labels are shown on the left, and correct labels on the right. Both random and true labels are trained using the same algorithm and hyperparameters on a fully connected ReLU network with three hidden layers of 500 (MNIST) or 1000 (CIFAR-10) units, followed by LeNet-5 (MNIST) or VGG-16 (CIFAR-10) shown in the subsequent row. The calibration factors for MNIST are 0.26 and 0.08, for CIFAR-10 0.24 and 0.18. The training error, test error, and our bound for the Gibbs posterior average of the 0–1 loss are plotted against $\beta$.
• Figure 5: ULA on MNIST and CIFAR-10 with 2000 training examples using BBCE loss function. The first row corresponds to MNIST and the second row to CIFAR-10. Random labels are shown on the left, correct labels on the right. Both random and true labels are trained with the same algorithm and parameters on a fully connected ReLU network with one (respectively two) hidden layers of 500 (respectively 1000) units. The calibration factor for MNIST is 0.49, for CIFAR-10 0.46. Train error, test error and our bound for the Gibbs posterior average of the 0-1 loss are plotted against $\beta$.

Theorems & Definitions (19)

• Lemma 3.1
• proof
• Theorem 3.2
• proof
• Corollary 3.3
• Theorem 4.1
• Corollary 4.2
• proof
• Theorem 4.3
• Theorem 4.4