Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FourierNAT: A Fourier-Mixing-Based Non-Autoregressive Transformer for Parallel Sequence Generation

Andrew Kiruluta, Eric Lundy, Andreas Lemos

TL;DR

FourierNAT proposes a non-autoregressive Transformer that inserts a Fourier-based mixing sub-layer in the decoder to propagate global context in parallel. By applying a discrete Fourier transform along the target sequence, gating the real and imaginary components, and performing an inverse transform, the model achieves global token mixing in a single pass, augmented by cross-attention to the encoder. The approach yields substantial decoding speedups over autoregressive baselines on WMT14 En--De and CNN/DailyMail while maintaining competitive quality against NAT methods, with optional refinement passes to further improve coherence. This spectral-domain strategy demonstrates the potential for fast, coherent parallel text generation and suggests future extensions to other transforms and larger-scale tasks.

Abstract

We present FourierNAT, a novel non-autoregressive Transformer (NAT) architecture that employs Fourier-based mixing in the decoder to generate output sequences in parallel. While traditional NAT approaches often face challenges with capturing global dependencies, our method leverages a discrete Fourier transform to mix token embeddings across the entire sequence dimension, coupled with learned frequency-domain gating. This allows the model to efficiently propagate context without explicit autoregressive steps. Empirically, FourierNAT achieves competitive results against leading NAT baselines on standard benchmarks like WMT machine translation and CNN/DailyMail summarization, providing significant speed advantages over autoregressive Transformers. We further demonstrate that learned frequency-domain parameters allow the model to adaptively focus on long-range or short-range dependencies, partially mitigating the well-known coherence gaps in one-pass NAT generation. Overall, FourierNAT highlights the potential of integrating spectral-domain operations to accelerate and improve parallel text generation. This approach can potentially provide great computational and time savings in inference tasks LLMs.

FourierNAT: A Fourier-Mixing-Based Non-Autoregressive Transformer for Parallel Sequence Generation

TL;DR

FourierNAT proposes a non-autoregressive Transformer that inserts a Fourier-based mixing sub-layer in the decoder to propagate global context in parallel. By applying a discrete Fourier transform along the target sequence, gating the real and imaginary components, and performing an inverse transform, the model achieves global token mixing in a single pass, augmented by cross-attention to the encoder. The approach yields substantial decoding speedups over autoregressive baselines on WMT14 En--De and CNN/DailyMail while maintaining competitive quality against NAT methods, with optional refinement passes to further improve coherence. This spectral-domain strategy demonstrates the potential for fast, coherent parallel text generation and suggests future extensions to other transforms and larger-scale tasks.

Abstract

We present FourierNAT, a novel non-autoregressive Transformer (NAT) architecture that employs Fourier-based mixing in the decoder to generate output sequences in parallel. While traditional NAT approaches often face challenges with capturing global dependencies, our method leverages a discrete Fourier transform to mix token embeddings across the entire sequence dimension, coupled with learned frequency-domain gating. This allows the model to efficiently propagate context without explicit autoregressive steps. Empirically, FourierNAT achieves competitive results against leading NAT baselines on standard benchmarks like WMT machine translation and CNN/DailyMail summarization, providing significant speed advantages over autoregressive Transformers. We further demonstrate that learned frequency-domain parameters allow the model to adaptively focus on long-range or short-range dependencies, partially mitigating the well-known coherence gaps in one-pass NAT generation. Overall, FourierNAT highlights the potential of integrating spectral-domain operations to accelerate and improve parallel text generation. This approach can potentially provide great computational and time savings in inference tasks LLMs.

Paper Structure

This paper contains 21 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Background and Introduction
  2. Prior Work with Partial Similarities
  3. FourierNAT Architecture
  4. Mathematical Formulation
  5. Notation and Encoder Output
  6. NAT Decoder Setup
  7. Training Objective
  8. Experiments: Settings and Parameters
  9. Data Preprocessing
  10. Model Architecture
  11. Optimization and Training Details
  12. Decoding and Refinement Passes
  13. Evaluation Metrics
  14. Summary
  15. Experimental Results and Discussion
  16. ...and 6 more sections

Figures (2)

  • Figure 1: High-level depiction of the proposed FourierNAT decoder block. After cross-attention, the hidden states go through a DFT-based mixing operation with learnable real/imag gating, enabling global context propagation without explicit autoregressive steps.
  • Figure 2: Left: Convergence of BLEU on WMT14 En--De over 200k steps. Right: Convergence of ROUGE-L on the CNN/DailyMail validation set over 160k steps. We compare FourierNAT (blue curve) with an AR Transformer (orange) and a standard NAT (green).