Revisiting Generalization Measures Beyond IID: An Empirical Study under Distributional Shift

TL;DR

This work evaluates whether generalization measures trained on IID data reliably predict performance under distribution shifts. By sweeping 40 measures across thousands of model configurations on CIFAR-10, DomainBed, and related benchmarks, the authors show that most measures lose predictive power under OOD, while calibration, information-criteria, and optimization-dynamics signals can become strong predictors in shifted environments. The study emphasizes that there is no single universally reliable metric, and that relying on a single proxy for model selection is risky. Practically, these findings advocate for diversified, regime-aware evaluation to guide robust deployment in non-stationary settings.

Abstract

Generalization remains a central yet unresolved challenge in deep learning, particularly the ability to predict a model's performance beyond its training distribution using quantities available prior to test-time evaluation. Building on the large-scale study of Jiang et al. (2020). and concerns by Dziugaite et al. (2020). about instability across training configurations, we benchmark the robustness of generalization measures beyond IID regime. We train small-to-medium models over 10,000 hyperparameter configurations and evaluate more than 40 measures computable from the trained model and the available training data alone. We significantly broaden the experimental scope along multiple axes: (i) extending the evaluation beyond the standard IID setting to include benchmarking for robustness across diverse distribution shifts, (ii) evaluating multiple architectures and training recipes, and (iii) newly incorporating calibration- and information-criteria-based measures to assess their alignment with both IID and OOD generalization. We find that distribution shifts can substantially alter the predictive performance of many generalization measures, while a smaller subset remains comparatively stable across settings.

Figures (9)

• Figure 1: Relationship between IID generalization-gap correlation and OOD generalization-gap correlation on CIFAR-10 suits. Each figure shows a different measure family.
• Figure 2: Sign-error distribution of NiN on CIFAR-10 suites. Figure \ref{['fig:nin_IID_distribution']} corresponds to the IID setting, and Figure \ref{['fig:nin_OOD_distribution']} corresponds to CIFAR-10-C.
• Figure 3: VLCS & PACS: Relationship between IID and OOD generalization-gap sensitivity ($\Psi$). Each panel shows a different measure family.
• Figure 5: Sign-error distribution of NiN on CIFAR-10-P
• Figure 6: Sign-error distributions of generalization-gap sensitivity for SimpleCNN