Sequential Group Composition: A Window into the Mechanics of Deep Learning

TL;DR

This work introduces the sequential group composition task as a principled probe of how neural networks learn to compose transformations from sequences. By embedding group elements with orbit-based encodings and leveraging the group Fourier transform, the authors derive a precise, mathematically grounded picture: two-layer nets learn irreducible representations one at a time in an order dictated by the encoding’s harmonic content, but require exponential width in the sequence length $k$ to achieve exact learning. Depth unlocks dramatic efficiency gains by exploiting associativity: RNNs perform sequential composition in $\mathcal{O}(k)$ steps, while multilayer MLPs realize parallel composition in $\mathcal{O}(\log k)$ layers, with Transformers potentially offering algebraic shortcuts. The Alternating Gradient Flow framework reveals a staged learning dynamic with distinct utility-maximization and cost-minimization phases, and the results establish a clear connection between representation learning, network architecture, and learning dynamics for sequential data. Overall, the paper provides a tractable, analytical window into how deep networks acquire structured, algebraic capabilities, with implications for understanding and designing architectures for algorithmic tasks.

Abstract

How do neural networks trained over sequences acquire the ability to perform structured operations, such as arithmetic, geometric, and algorithmic computation? To gain insight into this question, we introduce the sequential group composition task. In this task, networks receive a sequence of elements from a finite group encoded in a real vector space and must predict their cumulative product. The task can be order-sensitive and requires a nonlinear architecture to be learned. Our analysis isolates the roles of the group structure, encoding statistics, and sequence length in shaping learning. We prove that two-layer networks learn this task one irreducible representation of the group at a time in an order determined by the Fourier statistics of the encoding. These networks can perfectly learn the task, but doing so requires a hidden width exponential in the sequence length $k$. In contrast, we show how deeper models exploit the associativity of the task to dramatically improve this scaling: recurrent neural networks compose elements sequentially in $k$ steps, while multilayer networks compose adjacent pairs in parallel in $\log k$ layers. Overall, the sequential group composition task offers a tractable window into the mechanics of deep learning.

Figures (5)

• Figure 1: A unifying abstraction. Across arithmetic, perception, navigation, and planning, many sequence tasks require learning to compose transformations from examples. Motivated by this shared structure, we introduce the sequential group composition task—a unifying abstraction where networks learn to map a sequence of group elements to their cumulative product \ref{['eq:groupcomp']}.
• Figure 2: Visual introduction to abstract harmonic analysis. (a) The dihedral group $D_3$ consists of all rotations and reflections of a regular triangle, a canonical non-Abelian group where composition is order-dependent. (b) Its regular representation acts on $\mathbb{C}^{|G|}$ as $6 \times 6$ permutation matrices, which decompose into two one-dimensional and one two-dimensional irreducible representations (irreps). We encode $G$ by taking the orbit of a fixed encoding vector $x \in \mathbb{R}^6$ under the regular representation; this reduces to the standard one-hot encoding when $x = e_1$. (c) The Fourier transform is a unitary change of basis built from the irreps of $G$: see, e.g., how its first row corresponds to flattening the irreps of the identity element $1$. It decomposes a signal $x \in \mathbb{R}^{|G|}$ into its irrep components, with coefficients $\hat{x} = F^\dagger x$. This construction generalizes the classical DFT, recovered when $G = C_p$. Here we show the Fourier transform for $D_3$ and $C_6$.
• Figure 3: Binary composition on Abelian and Non-Abelian groups. A two-layer quadratic MLP learns to perform binary group composition task on Abelian and non-Abelian groups by learning the irreducible representations of the group one at a time in order of their importance to the encoding of the group as prescribed in \ref{['eq:utilitymaxshifts_new_main']}. Experimental details are given in \ref{['app:data']}.
• Figure 4: Two-layer networks need an exponential width. For $k=2$ (left) and $k=3$ (right), we report results from 400 training runs (20 group sizes $\times$ 20 hidden widths) with the cyclic group. Heatmap colors indicate training loss at convergence, defined as the network achieving a $99\%$ reduction in loss or exhausting the maximum allotted $10^9$ samples seen from the training distribution. The solid line show the theoretical lower bound for perfect learning, $H \geq (k+1)2^{k-1}|G|$, and the dashed lines delineates regions where the network has sufficient capacity to find partial solutions.
• Figure 5: Verifying dimensional bias in $D_3$. Power Spectrum components $\rho_1$ (1D) and $\rho_2$ (2D) during training across sequence lengths $k= 2, 3, 4, 5$ for the group $D_3$. The bias towards learning low-dimensional irreps first increases with $k$.

Theorems & Definitions (26)

• Definition 3.1
• Definition 3.2
• Definition 3.3
• Definition 3.4
• Lemma 3.5
• Theorem 4.1
• Theorem 4.3
• Lemma 1.1
• proof
• Lemma 2.1