FAN: Fourier Analysis Networks

TL;DR

General-purpose neural nets struggle to model periodic phenomena and generalize to out-of-domain data. This paper introduces FAN, a Fourier Analysis Network that embeds frequency components via a dedicated FAN layer $\phi(x)=[\cos(W_p x)||\sin(W_p x)||\sigma(B_{\bar p}+W_{\bar p}x)]$, with depth coordinating Fourier-coefficient capacity and intermediate-layer periodicity. Empirically, FAN achieves superior periodicity modeling and strong performance on real-world tasks (symbolic formula representation, time-series forecasting, language modeling) while using fewer parameters/FLOPs than MLP and outperforming Fourier-based networks in deeper settings. The results suggest FAN as a scalable, general-purpose periodicity-aware building block with potential to underpin large-scale foundation models and cross-domain applications.

Abstract

Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel neural network that effectively addresses periodicity modeling challenges while offering broad applicability similar to MLP with fewer parameters and FLOPs. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but face challenges in scaling to deeper networks and are typically designed for specific tasks, our approach overcomes this challenge to enable scaling to large-scale models and maintains general-purpose modeling capability. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks. Moreover, we reveal that compared to existing Fourier-based networks, FAN accommodates both periodicity modeling and general-purpose modeling well.

Figures (10)

• Figure 1: The performance of different neural networks within and outside the domain of their training data for the sine function, where $x$ is a scalar variable.
• Figure 2: Illustrations of FAN layer $\phi(x)$ vs. MLP layer $\Phi(x)$.
• Figure 3: The performance of FAN in periodicity modeling compared to MLP, KAN, and Transformer (Part I), where the green line represents the test data within the domain of training data, while the blue line represents the test data outside the domain of training data.
• Figure 4: Comparison of training and test losses for different models on the tasks of learning complex periodic functions.
• Figure 5: Comparison FAN with Fourier-based Networks on complex periodicity modeling ($y = e^{\sin(\pi x)^2 + \cos(x) + (x \mod 3) - 1}$) and language modeling.