Flow Equivariant Recurrent Neural Networks

TL;DR

This work introduces Flow Equivariance for sequence models, defining flow transformations as time-parameterized symmetries generated by one-parameter Lie subgroups. It then presents FERNN, a recurrent architecture that lifts hidden states to a velocity-augmented space V×G and uses flow-aware input lifting, convolutions, and recurrence to achieve true flow equivariance. Empirically, FERNNs exhibit zero-shot generalization to unseen flow velocities, improved length generalization, and superior performance on Flowing MNIST and moving-KTH action recognition compared with non-equivariant and static-equivariant baselines. The work discusses practical limitations (velocity-constant assumption, boundary truncation) and outlines future directions toward steerable flow equivariance and broader applicability to dynamic symmetry in sequences. Overall, this establishes a foundational framework for dynamic, time-parameterized symmetry in recurrent models with demonstrated gains in data efficiency and generalization under motion.

Abstract

Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.

Figures (16)

• Figure 1: Static vs. Flow Symmetries. Static symmetries are constant while flows progress over time.
• Figure 2: G-RNNs are not generally flow equivariant. We show this by simple counterexample with: $h_{t+1} = h_t + f_t$. See § \ref{['sec:GRNN_not_flow_eq']} for the full proof.
• Figure 2: FERNNs outperform non-equivariant models on sequence classification in the presence of strong motion. Test accuracy (mean $\pm$ std) for models trained and tested on the Moving KTH dataset ($V_2^T$).
• Figure 3: Increased flow equivariance increases training speed on data with flow symmetry. Validation loss vs. train steps.
• Figure 4: FERNNs exhibit next-step prediction length-generalization capabilities far surpassing G-RNNs on simple flows. We plot samples for the forward prediction trajectories of a G-RNN and FERNN-$V^{T}_2$ trained on Translating MNIST $V_2^T$ to predict 10-steps into the future (down-sampled by a factor of 4 in time for visualization). We see the G-RNN performs well on this training regime but diverges rapidly with lengths longer than training. The FERNN generalizes nearly perfectly. On the right, we plot this forward prediction error vs. the forward prediction timestep for both models.

Theorems & Definitions (17)

• Definition 3.1: Flow Equivariance
• Remark : Frame-wise feed-forward equivariant models are trivially flow equivariant.
• Theorem 3.1: G-RNNs are not flow equivariant
• Remark
• Theorem 4.1
• proof
• proof
• proof
• proof
• Lemma A.1