Learning and extrapolating scale-invariant processes

TL;DR

This work investigates extrapolating scale-free, self-similar processes using scale-equivariant neural operators. By integrating Fourier embedding, Fourier-Mellin, and wavelet-based building blocks (alongside a Riesz baseline), it analyzes spectral bias and cross-scale extrapolation in two physics-inspired tasks: a linear Fractional Gaussian Field and the Abelian Sandpile Model. The results show that Fourier-based architectures mitigate spectral bias and enable meaningful extrapolation across scales in FGFs, while FM-based approaches excel in nonlocal spectral-flow tasks; ASM remains challenging, underscoring the need for nontrivial coarse-grained representations. Collectively, the study provides a theoretical and empirical framework for designing scale-aware neural operators, with implications for modeling complex, self-similar systems and guiding future RG-inspired latent representations.

Abstract

Machine Learning (ML) has deeply changed some fields recently, like Language and Vision and we may expect it to be relevant also to the analysis of of complex systems. Here we want to tackle the question of how and to which extent can one regress scale-free processes, i.e. processes displaying power law behavior, like earthquakes or avalanches? We are interested in predicting the large ones, i.e. rare events in the training set which therefore require extrapolation capabilities of the model. For this we consider two paradigmatic problems that are statistically self-similar. The first one is a 2-dimensional fractional Gaussian field obeying linear dynamics, self-similar by construction and amenable to exact analysis. The second one is the Abelian sandpile model, exhibiting self-organized criticality. The emerging paradigm of Geometric Deep Learning shows that including known symmetries into the model's architecture is key to success. Here one may hope to extrapolate only by leveraging scale invariance. This is however a peculiar symmetry, as it involves possibly non-trivial coarse-graining operations and anomalous scaling. We perform experiments on various existing architectures like U-net, Riesz network (scale invariant by construction), or our own proposals: a wavelet-decomposition based Graph Neural Network (with discrete scale symmetry), a Fourier embedding layer and a Fourier-Mellin Neural Operator. Based on these experiments and a complete characterization of the linear case, we identify the main issues relative to spectral biases and coarse-grained representations, and discuss how to alleviate them with the relevant inductive biases.

Figures (22)

• Figure 1: Sample input and output with lattice size $L^2 = 64\times 64$ for the FGF spectral flow inference task. The transformation applied corresponds to a rescaling of factor $s=0.9$ and a rotation of $\pi/4$.
• Figure 2: Sample input and label with lattice size $32\times 32$ for the regression task in the ASM. In the input, the unstable site corresponding to the epicenter of the avalanche has a height of $5$. The output is a binary map, the ones marking the sites that toppled during the avalanche.
• Figure 3: Wavelet decomposition of the input. $\varphi_0$ can be perfectly recovered from $\varphi_2, \bar{\varphi}_2, \bar{\varphi}_1$. At each level, the number of "pixels" is divided by $4$.
• Figure 4: Graph representation of a $4\times 4$ input map, where we only show the edges outgoing from two nodes (highlighted in red). Each level in the tree corresponds to a scale of the wavelet decomposition (3 levels here). The 9 edge types are colored differently, in green for the child to parent relation, in shades of blue for the parent-children relations, and shades of orange/yellow for the neighbors relations. The node features $h_i$ contain the three wavelet coefficients and the value of the coarse-grained field at each position at the given scale. In dashed lines we show the edges that would continue if the tree continued downward.
• Figure 5: Normalized Mean Squared Error $E(|\mathbf k|)$ (equation (\ref{['eq:normalizedPS']})) evaluated on the test set for the Riesz network, the U-net, and the Fourier embedding network. The last one perfectly recovers the signal. The dashed gray line corresponds to the random predictor.