FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation

TL;DR

This work analyzes the frequency-domain properties of diffusion LLMs (dLLMs) and finds that low-frequency hidden-state components capture global structure while high-frequency components encode local details. Building on this, it introduces FourierSampler, a frequency-guided decoding strategy that uses a dynamic sliding window to move from structure to detail, via a Translated Filtering Score and an Adaptive Fourier Calibrator to balance guidance with model confidence. Across both fully bidirectional LLaDA and block-wise SDAR architectures, FourierSampler yields consistent performance gains on math and code tasks, surpassing similarly sized autoregressive models in several settings and achieving substantial improvements (e.g., up to 20.4% on MBPP and 45.1% on SDAR-1.7B-Chat). The results support an endogenous, frequency-based approach to unlock non-autoregressive potential in dLLMs and highlight the practical value of spectral-guided decoding for complex generation tasks.

Abstract

Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct.

Figures (7)

• Figure 1: The average score in different tasks of FourierSampler compared with other decoding strategies in LLaDA nie2025largezhu2025llada.
• Figure 2: Visualization of the correspondence between frequency-domain analysis and textual information. In the hidden states after a forward pass, tokens dominated by low-frequency signals correspond to structural information like if and elif, while tokens dominated by high-frequency signals correspond to detailed information like gcd.
• Figure 3: Overview of our FourierSampler. A sliding window in the frequency domain, retaining the low frequency at the beginning and the high frequency at the end based on the decoding step $s$, guides the dLLM to decode structural content first, then detailed content via Translated Fourier Score and Adaptive Fourier Calibrator.
• Figure 3: Results on LLaDA Series nie2025largezhu2025llada for the ablation study of FourierSampler.
• Figure 4: Low- and high-frequency features of different parts of speech in the frequency domain. The blue bars represent the proportion of tokens classified into the low group, while the orange bars represent the high group. Observations indicate that functional words responsible for syntactic structure, like conjunctions, exhibit dominant low-frequency features. In contrast, nouns, which typically serve as specific content fillers, show the strongest high-frequency tendency. This distribution corroborates our hypothesis that low-frequency components encode the structural skeleton, while high-frequency components correspond to detailed entities.