Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling

TL;DR

This work addresses the inefficiency of Transformer-based LLMs in learning periodic patterns by introducing FANformer, which integrates Fourier-based periodicity modeling into the attention mechanism via the ATtention-Fourier (ATF) module. The approach reshapes input features with frequency-domain representations and retains compatibility with standard attention through the equivalence ATF( X ) = Attention( FANLayer'( X ) ), enabling multi-head extensions and seamless integration with existing training techniques. Empirical results show FANformer scales more efficiently than Transformers, requiring roughly 0.692x the parameter count and 0.798x the training tokens to reach parity, and a pretrained FANformer-1B outperforms open-source LLMs of similar size on downstream tasks while using fewer data. Analyses reveal FANformer enhances rule-based and logical reasoning, increases representational capacity as evidenced by higher Lipschitz constants, and yields faster convergence in later training stages, underscoring its potential as a scalable alternative for advancing LLMs. Overall, FANformer demonstrates that effective periodicity modeling can substantially improve learning efficiency, scalability, and reasoning in large language models.

Abstract

Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which adapts Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. Our pretrained FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. Moreover, we reveal that FANformer exhibits superior ability to learn and apply rules for reasoning compared to Transformer. The results position FANformer as an effective and promising architecture for advancing LLMs.

Figures (13)

• Figure 1: The performance of FANformer on periodicity modeling and language modeling. (a) shows the training loss of Transformer and FANformer on the fitting mod function and their performance at the 4,000th epoch. (b) shows the average performance of 8 core commonsense tasks between FANformer-1B and the open-source LLMs with comparable model parameters and training tokens.
• Figure 2: Left: The illustration of FANformer's architecture. Right: The pseudocode of Multi-head ATF, where $p$ is the hyperparameter that controls the proportion of periodicity modeling for $\mathbf{X}_{p}$.
• Figure 3: Language modeling loss of scaling up model parameters and training tokens. Left: we train LLMs from 268M to 7B parameters. Right: we evaluate LLMs every 20B tokens up to 200B tokens.
• Figure 4: Training loss of FANformer and Transformer on early training steps, with the complete training loss is provided in Figure \ref{['CompleteTrainingDynamics']} of Appendix.
• Figure 5: Performance of FANformer and Transformer on modular addition and linear regression tasks, where the darkened regions indicate areas where the model performance approaches zero, signifying the emergence of the "hole" as described in the work CaseorRule-Based.

Theorems & Definitions (2)

• Proposition 1
• proof