Emergent weight morphologies in deep neural networks

TL;DR

The paper investigates whether deep neural networks develop emergent weight morphologies during training that do not depend on the training data. By formulating a path-activity framework inspired by condensed matter physics, it derives that the homogeneous weight state is unstable and gives rise to periodic channel structures, which are then corroborated by numerical experiments across synthetic and real datasets. The study further shows that channel amplitudes exhibit oscillatory modulation across layers and that embedding dimensions correlate with these morphologies, suggesting functional links to representation learning. Altogether, the work reveals universal, data-independent weight morphologies that constrain and potentially aid learning, with implications for pruning, representation, and security considerations in increasingly capable AI systems.

Abstract

Whether deep neural networks can exhibit emergent behaviour is not only relevant for understanding how deep learning works, it is also pivotal for estimating potential security risks of increasingly capable artificial intelligence systems. Here, we show that training deep neural networks gives rise to emergent weight morphologies independent of the training data. Specifically, in analogy to condensed matter physics, we derive a theory that predict that the homogeneous state of deep neural networks is unstable in a way that leads to the emergence of periodic channel structures. We verified these structures by performing numerical experiments on a variety of data sets. Our work demonstrates emergence in the training of deep neural networks, which impacts the achievable performance of deep neural networks.

Figures (8)

• Figure 1: Illustration of the theoretical approach. a As a first step, we define a unit describing the local morphology of weights (dashed rectangle). This unit is mathematically represented by the product of in- and outgoing weight fractions of a node. b We then infer effective interactions between these morphological units, represented by the arrows in this figure. The shading represents the value of the morphological unit. c We finally predict emergent, large-scale morphological structures from these interactions. Shading as in b.
• Figure 2: a Schematic depicting effective interactions between nodes in the same layer. b Numerical solution of Eq. \ref{['eq:intralayer']} using a 5th order Runge-Kutta scheme. The simulation uses 20 nodes in the layer and a uniform initial distribution for both the connectivities $r_i$ and constants $c_i$. Each simulation ran for 250 timesteps ($t_{\text{max}}=1.0$), and 1000 simulations were aggregated. c Schematic showing the mechanism leading to channel formation. d Snapshots of a neural network at different stages during early training. The shade of the connecting lines denotes the relative absolute strength of the weight with respect to the maximum within each layer. The neural network was trained on synthetic cluster data. Nodes in all images are ordered from top to bottom by the value of their connectivity after training. e Outgoing weight fraction $\Omega_{\text{out}}$ as a function of the ingoing weight fraction $\Omega_\text{in}$ for three different data sets. Each point corresponds to an individual node from 250 networks in total. Pearson correlation coefficient $r$ and $p$-value are shown. Dashed lines correspond to a linear fit through the origin. f Number of accessible nodes when traversing the network backwards from output to input, after pruning away all weights smaller in absolute value than the mean. Grey lines correspond to values computed before training, colored lines denote values computed after training. Layer difference 0 corresponds to the input layer. Shaded areas denote standard errors.
• Figure 3: a Schematic illustrating lateral interactions modulating the channel amplitude. b Amplitude increments between consecutive layers obtained from numerical solution of the connectivity dynamics with coupling to neighbouring layers using a 5th order Runge-Kutta scheme. The simulated network consisted of 10 nodes and 200 layers, and both initial connectivities and constants were drawn from a uniform distribution. The inlay shows an exemplary region with oscillating, anticorrelated amplitude increments. c Pearson autocorrelation of numerical simulations as in b. Each simulation consisted of 10 nodes and 12 layers, and both initial connectivities and constants are drawn from a uniform distribution. In total 250 simulations were aggregated. The shaded area denotes the standard error. d Snapshots of a neural network at different stages after channel formation until the end of training. The neural networks was trained on synthetic cluster data. Line shades and nodal permutation as in Fig. \ref{['fig:2']}d. e Pearson autocorrelation of amplitude increments as a function of the layer difference. Shaded areas denote standard errors.
• Figure 4: a Pearson correlation of in- and outgoing weight fractions as in Fig. \ref{['fig:2']}e as a function of the standard deviation of the initial weight distribution. b Fraction of accessible nodes, computed as the ratio between the colored line and the gray line in Fig. \ref{['fig:2']}f, as a function of the standard deviation of the initial weight distribution. c Number of accessible nodes for poorly trained networks with an accuracy below 20% before (gray line) and after (colored line) training, after pruning away all weights smaller in absolute value than the mean. Shaded areas denote standard errors. d Amplitude increments from one layer to the next scattered against the corresponding change in embedding dimension. Each point thus corresponds to a comparison of two successive layers. Pearson correlation coefficient $r$ and $p$-value are shown.
• Figure S1: The weight update of $w_{ab}^{(p)}$ consists of all paths through this weight. The possible connections from layer $p-2$ to $p-1$ and from $p$ to $p+1$ are therefore restricted to those into and out of nodes $a$ and $b$ as in this exemplary 3-node-network. We specify these restricted connections by the nodes $n$ and $n'$ connecting to $a$ and $b$. Outside of this subset of layers, we do not have to impose any restrictions on the paths. We capture this part of the weight update by $\mathcal{U}_{n}^{n'}$, all paths into node $n^{(p-2)}$ and out of node $n'^{(p+1)}$.