Softly Induced Functional Simplicity Implications for Neural Network Generalisation, Robustness, and Distillation

TL;DR

The paper addresses robust generalisation in high-dimensional HEP data by introducing a soft symmetry-inducing bias (SEAL) that steers optimisation toward low-complexity solutions. It quantifies functional complexity with Hessian-based metrics ($\mathrm{Tr}(\mathcal{H})$, $\lambda_1$) and with compressibility via distillation, revealing a mechanism termed a pseudo-Goldstone mode—a low-curvature degeneracy aligned with approximate symmetry. Empirically, constrained models maintain in-distribution accuracy while exhibiting reduced curvature, stronger robustness to input perturbations, better out-of-distribution generalisation, and faster, more efficient distillation compared with unconstrained baselines. This work suggests that high-capacity models can achieve transferable, distillable abstractions in collider tasks when guided by appropriate inductive biases, with implications for deployment in resource-constrained environments.

Abstract

Learning robust and generalisable abstractions from high-dimensional input data is a central challenge in machine learning and its applications to high-energy physics (HEP). Solutions of lower functional complexity are known to produce abstractions that generalise more effectively and are more robust to input perturbations. In complex hypothesis spaces, inductive biases make such solutions learnable by shaping the loss geometry during optimisation. In a HEP classification task, we show that a soft symmetry respecting inductive bias creates approximate degeneracies in the loss, which we identify as pseudo-Goldstone modes. We quantify functional complexity using metrics derived from first principles Hessian analysis and via compressibility. Our results demonstrate that solutions of lower complexity give rise to abstractions that are more generalisable, robust, and efficiently distillable.

Figures (9)

• Figure 1: Visualisation of loss geometry of ResNet-56 with skip connection (left) versus without (right). Wider minima correspond to reduced effective dimensionality. Figure taken from visualisaing_loss.
• Figure 2: Illustrating the connection between parameter-space curvature (left) and effective margins in input space (right). Leading Hessian eigenvectors correspond to directions that deform the decision boundary, while the number of significant modes controls the number of independent deformation directions.
• Figure 3: Hard and soft inductive biases and their effect on optimisation and generalisation. Without inductive bias (left), optimisation explores a large hypothesis space with no preference among solutions that fit the data, often leading to overfitting. Soft inductive biases (middle) preserve expressivity while biasing optimisation toward preferred regions of the loss geometry associated with better generalisation. Hard inductive biases (right) restrict the hypothesis space to enforce desired properties, reducing overfitting at the cost of reduced expressivity. Figure taken from andrew.
• Figure 4: Receiver operating characteristic (ROC) curves for the in-distribution Gluon vs W boson classification task, comparing unconstrained and constrained models. Both models achieve comparable discrimination performance.
• Figure 5: Loss geometry along the leading Hessian eigenvector $\nu_1$ for the constrained (left) and unconstrained (right) models. The constrained model exhibits a pseudo-Goldstone mode aligned with the approximate symmetry, whereas the unconstrained model shows no such degeneracy.