Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning
Sebastiano Ariosto
TL;DR
This work studies how statistical-physics methods illuminate deep neural networks, addressing generalization beyond infinite width, finite-width effects, and feature learning. It introduces three complementary avenues: a data-averaged bound showing generalization can depend primarily on the last-layer size, a data-dependent proportional-regime analysis linking DNNs to Gaussian and Student's t-processes, and a task-explicit study of whether networks internalize data structure or merely memorize. The data-averaged approach leverages quenched averages and replica methods to produce a tighter asymptotic bound Δε̃ ≤ 2 N_out / P, highlighting distinct layer roles and suggesting smaller last-layer width may improve generalization in some regimes. The proportional-regime analysis develops NN priors beyond IW, deriving closed-form or approximate generalization expressions for shallow and deep nets, and revealing a link to Student's t-processes, thereby bridging GP-based and finite-width descriptions. Finally, the task-explicit chapter investigates when DNNs truly learn data structure versus memorize, showing how optimal weight configurations emerge in underparameterized settings and how Gaussianity breaking accompanies feature learning. Collectively, the thesis advances a physics-informed, multi-regime understanding of when and how DNNs generalize, offering insights for architecture design and interpretability with potential practical impact in robust AI systems.
Abstract
Deep Neural Networks (DNNs) excel at many tasks, often rivaling or surpassing human performance. Yet their internal processes remain elusive, frequently described as "black boxes." While performance can be refined experimentally, achieving a fundamental grasp of their inner workings is still a challenge. Statistical Mechanics has long tackled computational problems, and this thesis applies physics-based insights to understand DNNs via three complementary approaches. First, by averaging over data, we derive an asymptotic bound on generalization that depends solely on the size of the last layer, rather than on the total number of parameters -- revealing how deep architectures process information differently across layers. Second, adopting a data-dependent viewpoint, we explore a finite-width thermodynamic limit beyond the infinite-width regime. This leads to: (i) a closed-form expression for the generalization error in a finite-width one-hidden-layer network (regression task); (ii) an approximate partition function for deeper architectures; and (iii) a link between deep networks in this thermodynamic limit and Student's t-processes. Finally, from a task-explicit perspective, we present a preliminary analysis of how DNNs interact with a controlled dataset, investigating whether they truly internalize its structure -- collapsing to the teacher -- or merely memorize it. By understanding when a network must learn data structure rather than just memorize, it sheds light on fostering meaningful internal representations. In essence, this thesis leverages the synergy between Statistical Physics and Machine Learning to illuminate the inner behavior of DNNs.