A Function Centric Perspective On Flat and Sharp Minima

TL;DR

The paper interrogates the long-held belief that flat loss minima reliably improve generalisation by adopting a function-centric view: the geometry of a minimum reflects the complexity of the learned function rather than universal generalisation quality. Through toy single-objective problems and large-scale vision tasks with regularisers such as Sharpness-Aware Minimisation (SAM), weight decay, and data augmentation, it shows that sharper minima can coincide with better generalisation, calibration, and robustness. The authors define and use multiple, reparameterisation-invariant sharpness metrics alongside safety evaluations (ECE, corruption robustness, and functional disagreement) to demonstrate that regularisation often yields sharper but more effective solutions, challenging the notion that flatness is universally preferable. They argue that sharpness captures task complexity and inductive bias, advocating for a function-centric interpretation of loss landscape geometry and its implications for safety-critical deployment of deep models.

Abstract

Flat minima are widely believed to correlate with improved generalisation in deep neural networks. However, this connection has proven more nuanced in recent studies, with both theoretical counterexamples and empirical exceptions emerging in the literature. In this paper, we revisit the role of sharpness in model performance, proposing that sharpness is better understood as a function-dependent property rather than a reliable indicator of poor generalisation. We conduct extensive empirical studies, from single-objective optimisation to modern image classification tasks, showing that sharper minima often emerge when models are regularised (e.g., via SAM, weight decay, or data augmentation), and that these sharp minima can coincide with better generalisation, calibration, robustness, and functional consistency. Across a range of models and datasets, we find that baselines without regularisation tend to converge to flatter minima yet often perform worse across all safety metrics. Our findings demonstrate that function complexity, rather than flatness alone, governs the geometry of solutions, and that sharper minima can reflect more appropriate inductive biases (especially under regularisation), calling for a function-centric reappraisal of loss landscape geometry.

Figures (11)

• Figure 1: Scatter plots of 240 converged minima for ResNet-18 on CIFAR-10 across batch size ${128, 256}$ and learning rate ${10^{-3},10^{-2}}$: (a) Fisher–Rao norm vs. train loss, (b) Relative Flatness vs. train loss, and (c) generalisation gap vs. train loss (log scale). Full results in Appendix \ref{['sec:ResNet_param_sweep']}.
• Figure 2: Himmelblau's function landscape with four global minima in red.
• Figure 3: Landscapes for six single-objective functions.
• Figure 4: Training an MLP on single-objective problems over epochs: mean sharpness, training loss, and absolute generalisation gap (averaged over 10 runs).
• Figure 5: Scatter plots an MLP trained on the sphere, rosenbrock, rastrigin, beale, booth, three-hump camel, and himmelblaus functions for 10 different datasets till reaching a target train loss of 300: (left) mean sam sharpness vs. train loss, (centre) $|$ generalisation gap $|$ vs. train loss, and (right) $|$generalisation gap$|$ vs. mean sam sharpness.