Do Transformers Have the Ability for Periodicity Generalization?

TL;DR

This work defines periodicity as invariance under transformations and frames it with group theory to distinguish sequence and rule periodicity, then extends to composite periodicity. It introduces the Coper dataset to evaluate interpolation (Hollow) and extrapolation (Extrapolation) in composite periodic tasks and reveals that current Transformer-based models, including RoPE-equipped variants and FANformer, can memorize seen periods but fail to generalize to unseen composite periodic rules. Key findings show RoPE can capture simple sequence periodicity but cannot generalize rule periodicity or composite periodicity; increasing data density and model size improves Hollow performance but has limited impact on Extrapolation. The study highlights the need for new architectures or inductive biases to achieve robust OOD periodicity generalization, with broad implications for reasoning and long-horizon generalization in AI systems, and provides a scalable benchmark for future research.

Abstract

Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research.

Figures (20)

• Figure 1: Accuracy heatmaps of Transformer and FANformer on composite periodicity tasks in Coper. Models perform well on seen composite periods, but accuracy drops in Hollow and Extrapolation OOD settings, showing limits in periodicity generalization. The full results are shown in Figure \ref{['fig:RQ1_full_heatmaps']}.
• Figure 2: Top: Transformer with RoPE generalizes well for transformations satisfying translation invariance ($f(t+T)=f(t)$). Bottom: Single-period sequences under a non-invariant transformation ($f(t+T)=2\cdot f(t)$), showing the generalization failure.
• Figure 3: Accuracy by category on composite periodicity tasks.
• Figure 4: Full accuracy heatmaps across all baseline models on composite periodicity tasks, corresponding to the final models trained as in Figure \ref{['fig:loss_curve']}. Notably, even when trained for 1000 epochs, RWKV and Mamba still fail to fit.
• Figure 5: Loss curves on ID and OOD testsets during training.