Critical Organization of Deep Neural Networks, and p-Adic Statistical Field Theories
W. A. Zúñiga-Galindo
TL;DR
This work provides a rigorous, p-adic, hierarchical framework for analyzing the thermodynamic limit of sigmoidal deep and recurrent neural networks, linking neural dynamics to aspects of statistical-field theories. It proves, under a contraction condition on activation Lipschitz constant and weight norms, the existence and continuous dependence of a unique hidden state in the thermodynamic limit; crossing a critical threshold yields a bifurcation to infinitely many states organized in a hierarchical lattice, interpreted as a strange-attractor-like organization in the $p$-adic setting. The paper further shows universality: any standard DNN/RNN can be recast as a $p$-adic discrete DNN with a tree-like index structure, and it develops a formalism for random infinite-width networks where outputs follow Gaussian processes with covariances determined by network kernels and activation nonlinearities. A detailed toy-model study and a complete path-integral formulation establish a formal DNN–SFT correspondence, providing a mathematically rigorous bridge between hierarchical NN architectures and ultrametric/statistical-field descriptions with potential implications for understanding deep learning dynamics and brain-inspired computation. Overall, the results offer a principled view of how hierarchical, critical, and stochastic aspects of DNNs can be modeled within a unified $p$-adic and field-theoretic framework, with concrete algorithms for constructing universal, tree-structured NN architectures and for quantifying their output distributions in the infinite-width limit.
Abstract
We rigorously study the thermodynamic limit of deep neural networks (DNNS) and recurrent neural networks (RNNs), assuming that the activation functions are sigmoids. A thermodynamic limit is a continuous neural network, where the neurons form a continuous space with infinitely many points. We show that such a network admits a unique state in a certain region of the parameter space, which depends continuously on the parameters. This state breaks into an infinite number of states outside the mentioned region of parameter space. Then, the critical organization is a bifurcation in the parameter space, where a network transitions from a unique state to infinitely many states. We use p-adic integers to codify hierarchical structures. Indeed, we present an algorithm that recasts the hierarchical topologies used in DNNs and RNNs as p-adic tree-like structures. In this framework, the hierarchical and the critical organizations are connected. We study rigorously the critical organization of a toy model, a hierarchical edge detector for grayscale images based on p-adic cellular neural networks. The critical organization of such a network can be described as a strange attractor. In the second part, we study random versions of DNNs and RNNs. In this case, the network parameters are generalized Gaussian random variables in a space of quadratic integrable functions. We compute the probability distribution of the output given the input, in the infinite-width case. We show that it admits a power-type expansion, where the constant term is a Gaussian distribution.