Draft-Conditioned Constrained Decoding for Structured Generation in LLMs

TL;DR

Draft-Conditioned Constrained Decoding is proposed, a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.

Abstract

Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose \emph{Draft-Conditioned Constrained Decoding (DCCD)}, a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative "projection tax" induced by hard constraints, with an optional best-of-$K$ draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2\% to 39.0\% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.

Figures (13)

• Figure 1: Our proposed approach, Draft-Conditioned Constrained Decoding (DCCD), yields consistent accuracy improvements over standard constrained decoding (state of the art) across model scales (1B--14B). Purple bars denote baseline constrained decoding accuracy, while green segments show the absolute accuracy gain from DCCD. These gains reflect improved response correctness and structure adherence at all model sizes.
• Figure 2: Low feasible mass results in distortion. Across tokens in a toy example, the feasible mass $\alpha(h_t)$ under constrained decoding is always $<0.53$ and is near zero for early tokens. In this setting, the constraint admits only $\approx 1\%$ of the full vocabulary as feasible tokens, forcing strong renormalization and accumulating KL distortion.
• Figure 3: Token-wise confidence distribution for answer tokens in a single example. Constrained decoding spreads probability mass across multiple plausible answer tokens ("$6$", "$27$", "$28$", "$6$", "$84$", "$9$"), with the incorrect answer "$27$" receiving moderate confidence (0.46). DCCD shows a sharp, concentrated distribution with near-perfect confidence (1.0) on the correct token "$14$".
• Figure 4: Average performance comparison across all evaluation datasets (GSM8K, GSM Symbolic, Math500, and FOLIO). We compare prompting-based baselines (CP, CF), grammar-based constrained decoding (CD), and our Draft-Conditioned Constrained Decoding (DCCD). Across all model scales, DCCD achieves the best aggregated performance, with the largest relative gains for smaller models where hard constraints induce the strongest projection distortion (e.g., 1B: 10.2$\%\rightarrow$20.9$\%$). Takeaway: conditioning on an unconstrained draft before enforcing hard constraints yields consistent, model-agnostic improvements in strict structured generation.
• Figure 5: Parameter efficiency (accuracy per billion parameters). For each dataset, we report strict structured accuracy normalized by total parameters for single-model CD and parameter-matched DCCD compositions. DCCD consistently achieves higher accuracy per parameter, with the largest gains in low-capacity regimes.