WaveFormer: Frequency-Time Decoupled Vision Modeling with Wave Equation

TL;DR

This paper tackles the limitation of attention-based and heat-based vision models in propagating semantic information across spatial frequencies. It introduces a physics-inspired framework where semantic propagation follows the underdamped wave equation, yielding a frequency–time decoupled solution implemented as the Wave Propagation Operator ($O(N \log N)$) and integrated into the WaveFormer backbone. The WaveFormer family achieves competitive ImageNet accuracy and superior throughput while reducing FLOPs relative to attention-based models, demonstrated across image classification, object detection, and semantic segmentation (e.g., WaveFormer-Base reaches 84.2% Top-1 on ImageNet-1K with notable efficiency gains). The results indicate that wave-based propagation provides a complementary bias to heat-based methods, preserving both global coherence and high-frequency details essential for rich visual semantics, with practical implications for efficient, interpretable vision backbones.

Abstract

Vision modeling has advanced rapidly with Transformers, whose attention mechanisms capture visual dependencies but lack a principled account of how semantic information propagates spatially. We revisit this problem from a wave-based perspective: feature maps are treated as spatial signals whose evolution over an internal propagation time (aligned with network depth) is governed by an underdamped wave equation. In this formulation, spatial frequency-from low-frequency global layout to high-frequency edges and textures-is modeled explicitly, and its interaction with propagation time is controlled rather than implicitly fixed. We derive a closed-form, frequency-time decoupled solution and implement it as the Wave Propagation Operator (WPO), a lightweight module that models global interactions in O(N log N) time-far lower than attention. Building on WPO, we propose a family of WaveFormer models as drop-in replacements for standard ViTs and CNNs, achieving competitive accuracy across image classification, object detection, and semantic segmentation, while delivering up to 1.6x higher throughput and 30% fewer FLOPs than attention-based alternatives. Furthermore, our results demonstrate that wave propagation introduces a complementary modeling bias to heat-based methods, effectively capturing both global coherence and high-frequency details essential for rich visual semantics. Codes are available at: https://github.com/ZishanShu/WaveFormer.

Figures (4)

• Figure 1: (a) The self-attention operator facilitates information from a pixel to all other pixels, resulting in $\mathcal{O}(N^2)$ complexity.(b) The wave propagation operator (WPO) introduces oscillatory dynamics that balance energy across different frequency components, enabling new modeling behavior and reduced complexity.
• Figure 2: The network architecture of WaveFormer. (a) The network follows a hierarchical vision backbone design, consisting of a stem followed by four stages of WPO Blocks, where each stage integrates the Wave Propagation Operator (WPO) with feed-forward layers and downsampling in between. (b) Implementation of the WPO. The input feature map is transformed into the frequency domain, where the frequency–time decoupled analytical solution of the underdamped wave equation modulates each frequency component through oscillatory dynamics. The result is then mapped back to the spatial domain via the inverse Fourier transform, enabling global semantic propagation while preserving fine-grained high-frequency details.
• Figure 3: Attention map evolution over time for heat conduction (top) and wave propagation (bottom) across different ADE20K image cases. Red boxes highlight key different regions.
• Figure 4: Evaluation of thermal wave velocity and damping term using WaveFormer-B with different $(v, \alpha)$ settings.