Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral-Window Hybrid (SWH)

Vladimer Khasia

TL;DR

The paper tackles the quadratic scaling of global self-attention in Transformers for long-context sequence modeling. It introduces Spectral-Window Hybrid (SWH), a parallel architecture that runs a global spectral convolution and a local chunked sliding-window attention, then fuses their outputs to maintain short-context accuracy while enabling long-context scalability. The global branch uses a learnable damped harmonic oscillator kernel implemented via FFT to achieve $O(T \log T)$ time, while the local branch employs RoPE and chunked attention with a 2W receptive field. Experiments on synthetic tasks and the FineWeb-Edu corpus show that SWH matches Transformer perplexity on short contexts, provides superior length generalization, and scales linearly in latency and memory for long sequences, highlighting its practical value for long-context language modeling; code is available at the provided GitHub link.

Abstract

Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity. While Transformers provide precise retrieval via the attention mechanism, their quadratic $\mathcal{O}(T^2)$ complexity limits their application to long-horizon tasks. In this work, we propose the \textbf{Spectral-Window Hybrid (SWH)}, an architecture that decouples sequence modeling into two \textit{parallel} streams: a global branch utilizing the Convolution Theorem to model long-range decay dynamics in $\mathcal{O}(T \log T)$ time, and a local branch employing sliding-window attention for token interactions within a bounded context. By aggregating these representations, SWH avoids the computational bottleneck of global attention while retaining local precision. We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences. The code is available at https://github.com/VladimerKhasia/SWH

Spectral-Window Hybrid (SWH)

TL;DR

The paper tackles the quadratic scaling of global self-attention in Transformers for long-context sequence modeling. It introduces Spectral-Window Hybrid (SWH), a parallel architecture that runs a global spectral convolution and a local chunked sliding-window attention, then fuses their outputs to maintain short-context accuracy while enabling long-context scalability. The global branch uses a learnable damped harmonic oscillator kernel implemented via FFT to achieve time, while the local branch employs RoPE and chunked attention with a 2W receptive field. Experiments on synthetic tasks and the FineWeb-Edu corpus show that SWH matches Transformer perplexity on short contexts, provides superior length generalization, and scales linearly in latency and memory for long sequences, highlighting its practical value for long-context language modeling; code is available at the provided GitHub link.

Abstract

Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity. While Transformers provide precise retrieval via the attention mechanism, their quadratic complexity limits their application to long-horizon tasks. In this work, we propose the \textbf{Spectral-Window Hybrid (SWH)}, an architecture that decouples sequence modeling into two \textit{parallel} streams: a global branch utilizing the Convolution Theorem to model long-range decay dynamics in time, and a local branch employing sliding-window attention for token interactions within a bounded context. By aggregating these representations, SWH avoids the computational bottleneck of global attention while retaining local precision. We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences. The code is available at https://github.com/VladimerKhasia/SWH
Paper Structure (17 sections, 9 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 17 sections, 9 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Global Branch: Causal Spectral Convolution
  4. Efficient Implementation via FFT
  5. Local Branch: Chunked Sliding-Window Attention
  6. Rotary Position Embeddings
  7. Context Management and Masking
  8. Algorithm
  9. Complexity Analysis
  10. Experiments
  11. Preliminary Synthetic Evaluation
  12. Language Modeling on FineWeb-Edu
  13. Computational Efficiency
  14. Conclusion
  15. Experimental Details
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Synthetic Task Performance. Comparison of Standard Attention vs. SWH on five algorithmic primitives. SWH demonstrates superior generalization on out-of-distribution lengths (LenGen, Needle) while maintaining strong performance on in-distribution tasks.
  • Figure 2: Validation Loss on FineWeb-Edu. The SWH model (Orange) consistently achieves lower loss compared to the Standard Transformer Baseline (Blue). Smoothed curves (solid lines) are overlaid on raw loss values (transparent).
  • Figure 3: Efficiency Analysis. Left: Inference latency (seconds) vs. sequence length. Right: VRAM usage (MB) vs. sequence length. SWH displays linear scaling, whereas the baseline exhibits quadratic growth, making SWH significantly more efficient for long sequences.