Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations

TL;DR

The paper investigates how symmetry constraints in neural networks interact with their expressivity when learning renormalisation-group–like decimation maps, using the central limit theorem as a controlled test case. It develops a cumulant-propagation framework to analyse how distributions transform through MLPs and extends this to GNNs, revealing a tension: overly symmetric or overly expressive models can generalise poorly. analytically shows that symmetrically constrained weights can overconstrain simple MLPs with nonlinear activations (e.g., quadratic) and hinder learning, while allowing asymmetry or limited nonlinearity can improve generalisation on CLT-inspired tasks. Empirically, the study demonstrates that the task structure dictates whether symmetry constraints help or hurt, with spline and GNN biases often acting as Occam’s razor toward minimal nonlinearity. The findings offer guidance for encoding physical priors in neural models and suggest adaptive symmetry approaches for robust learning in physics-inspired tasks.

Abstract

Deep learning models have proven enormously successful at using multiple layers of representation to learn relevant features of structured data. Encoding physical symmetries into these models can improve performance on difficult tasks, and recent work has motivated the principle of parameter symmetry breaking and restoration as a unifying mechanism underlying their hierarchical learning dynamics. We evaluate the role of parameter symmetry and network expressivity in the generalisation behaviour of neural networks when learning a real-space renormalisation group (RG) transformation, using the central limit theorem (CLT) as a test case map. We consider simple multilayer perceptrons (MLPs) and graph neural networks (GNNs), and vary weight symmetries and activation functions across architectures. Our results reveal a competition between symmetry constraints and expressivity, with overly complex or overconstrained models generalising poorly. We analytically demonstrate this poor generalisation behaviour for certain constrained MLP architectures by recasting the CLT as a cumulant recursion relation and making use of an established framework to propagate cumulants through MLPs. We also empirically validate an extension of this framework from MLPs to GNNs, elucidating the internal information processing performed by these more complex models. These findings offer new insight into the learning dynamics of symmetric networks and their limitations in modelling structured physical transformations.

Figures (18)

• Figure 1: (a) Network analysis based on data samples considers a direct mapping between samples $x$ and outputs $y$ comprised of affine and nonlinear transformations. (b) The alternative statistical viewpoint considers how networks transform the entire data distribution $p(x)$. Affine layers propagate cumulants independently but the nonlinearity mixes cumulants of different orders $n$. A Gaussian input with only non-zero mean $\mu$ and covariance $\Sigma$ is shown. Figure taken from fischer_decomposing_2022.
• Figure 2: Overview of how a single node aggregates messages for a two-layer model. The GNN computation graph forms a tree structure as messages are sequentially aggregated from each node's local neighbourhoods. Figure taken from hamilton2020graph.
• Figure 3: Normalised KL divergence $\hat{D}_{KL}(p_\text{net}\|p_\text{gt})$ as a function of input distribution variance $\sigma^2$ for linear networks. Error bars were negligible and are omitted.
• Figure 4: (a) Normalised KL divergence $\hat{D}_{KL}(p_\text{net}\|p_\text{gt})$ as a function of input variance $\sigma^2$. (b) Fractional deviations of second order cumulants $\kappa_2$ between network outputs and ground truth, expressed as percentages, as a function of variance $\sigma^2$. The three conditions shown all use unconstrained asymmetric weights. Error bars were negligible in (a) and are omitted.
• Figure 5: (a), (b) Normalised KL divergence $\hat{D}_{KL}(p_\text{net}\|p_\text{gt})$ as a function of input variance $\sigma^2$. (c), (d) Relative deviations of second order cumulants $\kappa_2$, expressed as percentages, as a function of variance $\sigma^2$. Left hand panels show all three activation types (linear, ReLU, quadratic); right hand panels show only linear networks and quadratic activations. All networks use symmetrically-constrained weights.