State Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling

TL;DR

The paper introduces a discrete diffusion-based language model, SFDLM, that avoids transformers by integrating time-domain state-space dynamics with a Complex Fourier MLP for global mixing. Tokens are progressively corrupted with a forward process using a time-dependent schedule, and a U-Net–style denoiser combines local SSM kernels with frequency-domain processing to reconstruct coherent text, enabling inpainting and partial editing. Empirical results on PTB, WikiText-103, and C4 show competitive perplexities at moderate scales and demonstrate the model’s iterative refinement capabilities, though it generally lags the best Transformer LLMs on standard benchmarks. The work highlights favorable scaling with sequence length and presents avenues for improvement, including improved noise schedules, hierarchical diffusion, and RLHF integration, suggesting a viable path toward scalable, editable language models beyond self-attention.

Abstract

In recent years, diffusion based methods have emerged as a powerful paradigm for generative modeling. Although discrete diffusion for natural language processing has been explored to a lesser extent, it shows promise for tasks requiring iterative denoising of token based data. In standard approaches to text generation, transformers dominate, but their reliance on self attention often incurs high computational costs. This paper introduces a fully diffusion driven discrete text generation model built without any transformer or large convolution modules. Instead, the model integrates structured state space dynamics in the time domain with a novel Complex Fourier Multi Layer Perceptron module that operates in the frequency domain. The forward noising process randomly samples the vocabulary to replace tokens with a controlled probability, while the learned reverse model systematically reverts corrupted sequences toward their original states. By composing local state space updates with global Fourier based mixing, the approach effectively captures both short and long range dependencies.

Figures (1)

• Figure 1: This figure illustrates the dual-path architecture of our discrete diffusion-based language model. On the left, the forward diffusion process begins with a clean token sequence $(x^0)$ that is progressively corrupted through a series of diffusion steps. At each step t, tokens are randomly replaced with other vocabulary tokens according to an increasing probability $\beta_t$, resulting in a sequence that gradually transitions from $x^0$(clean text) to $x^T$ (a nearly random token sequence). The reverse denoising path is depicted using a U-Net–style architecture that reconstructs the original text. This reverse process starts from a partially noised sequence $x^{(t+1)}$ (derived from the forward process) and applies token embeddings combined with time-step embeddings to signal the current noise level. The embedded sequence then flows through successive Down Blocks, each incorporating a state-space module for capturing local time-domain correlations and a Complex Fourier MLP for global frequency-domain mixing followed by a bottleneck and corresponding Up Blocks, which employ skip connections to merge fine and coarse information. Finally, a vocabulary projection layer outputs, logits, from which the denoised sequence $x^t$ is produced. Together, these forward and reverse paths enable iterative refinement and reconstruction, offering efficient scaling and direct text inpainting capabilities without relying on self-attention.