(G)I-DLE: Generative Inference via Distribution-preserving Logit Exclusion with KL Divergence Minimization for Constrained Decoding

TL;DR

Constrained decoding often bans tokens by setting logits to $-\infty$, which distorts the autoregressive distribution and can increase output variance in multilingual contexts. The authors reformulate constraint handling as a KL-divergence minimization problem and derive a distribution-preserving logit processor that renormalizes probabilities over allowed tokens, yielding $Q(x_t) = \frac{P(x_t \mid x_{<t})}{\sum_{j \notin B} P(j \mid x_{<t})}$ for $x_t \notin B$. They validate the approach on the $K^2$-Eval Korean evaluation dataset using Qwen2.5 models, demonstrating higher mean scores and substantially reduced variance compared to naive masking and baseline generation. This work provides a general framework for distribution-preserving constrained decoding with practical benefits for domain-specific, culturally aware generation in multilingual language models.

Abstract

We propose (G)I-DLE, a new approach to constrained decoding that leverages KL divergence minimization to preserve the intrinsic conditional probability distribution of autoregressive language models while excluding undesirable tokens. Unlike conventional methods that naively set banned tokens' logits to $-\infty$, which can distort the conversion from raw logits to posterior probabilities and increase output variance, (G)I-DLE re-normalizes the allowed token probabilities to minimize such distortion. We validate our method on the K2-Eval dataset, specifically designed to assess Korean language fluency, logical reasoning, and cultural appropriateness. Experimental results on Qwen2.5 models (ranging from 1.5B to 14B) demonstrate that G-IDLE not only boosts mean evaluation scores but also substantially reduces the variance of output quality.