Causal Autoregressive Diffusion Language Model

TL;DR

CARD addresses the inefficiency of autoregressive decoding and the latency of diffusion-based generation by reformulating diffusion within a strictly causal, GPT-like Transformer. It introduces soft tail masking and context-aware reweighting to stabilize training under causal diffusion, and leverages KV caching with a confidence-based dynamic inference strategy to achieve parallel decoding. Empirically, CARD matches or surpasses diffusion baselines on downstream tasks and perplexity, while delivering substantial training and inference speedups, thereby offering a data-efficient, scalable backbone for next-generation LLMs. This framework demonstrates a practical path to combining the stability and data efficiency of ARMs with the throughput advantages of diffusion, with significant implications for efficient, large-scale language modeling.

Abstract

In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion models. CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass. To address the optimization instability of causal diffusion, we introduce a soft-tailed masking schema to preserve local context and a context-aware reweighting mechanism derived from signal-to-noise principles. This design enables dynamic parallel decoding, where the model leverages KV-caching to adaptively generate variable-length token sequences based on confidence. Empirically, CARD outperforms existing discrete diffusion baselines while reducing training latency by 3 $\times$ compared to block diffusion methods. Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation, establishing a robust paradigm for next-generation efficient LLMs.

Figures (4)

• Figure 1: Comparison of training paradigms. Current diffusion methods like MDLM and BD3LM are inefficient compared to ARM; MDLM reaches only 50% of ARM's expected efficiency, while BD3LM relies on complex masking and sequence duplication. CARD overcomes these issues by using causal diffusion, maintaining the same high efficiency as ARM while achieving better performance.
• Figure 2: Inference comparison of the four paradigms. CARD achieves high-quality results similar to ARM. With KV cache support, friendly operators, and parallel generation, it offers faster throughput than earlier methods. In particular, our inference parallelism is flexible, unlike BD3LM which is tied to the fixed block size used during training.
• Figure 3: CARD Training Framework
• Figure 4: HellaSwag performance of four paradigms under repeated training on a FineWeb-Edu subset. The annotations mark specific crossover points in performance: P1 denotes the epoch where CARD surpasses ARM, P2 where MDLM overtakes ARM, and P3 where MDLM exceeds CARD.

Theorems & Definitions (7)

• Definition 1.1: Local Ambiguity Score
• Proposition 1.2
• proof
• Proposition 1.3
• proof
• Proposition 1.4
• proof