Improving Open-Ended Text Generation via Adaptive Decoding

TL;DR

This work introduces adaptive decoding, a method that dynamically determines the next-token candidate set for open-ended text generation by tracking the incremental increase in a normalized entropy-based confidence, $\text{Conf}_k(X)$. By iteratively transferring tokens with high probability into a growing candidate set and stopping when the confidence gain falls below a threshold $\epsilon$, the approach balances diversity and coherence while mitigating degeneration phenomena such as repetition and incoherence. Empirical results across GPT2-XL and Llama2-7B family models show improved MAUVE and diversity, with human evaluations indicating more human-preferred outputs; the method also yields gains on policy-tuned models and provides potential benefits for reasoning tasks in smaller models. The work demonstrates that entropy-based, adaptive truncation can offer robust improvements over traditional decoding schemes, albeit with extra computational overhead that can be mitigated with parallelization.

Abstract

Current language models decode text token by token according to probabilistic distribution, and determining the appropriate candidates for the next token is crucial to ensure generation quality. This study introduces adaptive decoding, a mechanism that dynamically empowers language models to ascertain a sensible candidate set during generation. Specifically, we introduce an entropy-based metric called confidence and conceptualize determining the optimal candidate set as a confidence-increasing process. The rationality of including a token in the candidate set is assessed by leveraging the increment of confidence. Experimental results reveal that our method balances diversity and coherence well. The human evaluation shows that our method can generate human-preferred text. Additionally, our method can potentially improve the reasoning ability of language models.

Figures (6)

• Figure 1: Human written text: "Barack Obama was born in 1961. He was raised in Hawaii by his mother and grandparents. Growing up, Obama faced the challenges of being biracial, with a Kenyan father and an American mother. Despite these challenges, he excelled academically and eventually attended Columbia University in New York City." We provide this human-written text for GPT2-XL and use teacher-forcing decoding.
• Figure 2: Preliminary experiment: Averaged top-20 token distribution of different LMs on 1000 human-written texts from WikitText-103.
• Figure 3: How adaptive decoding works. Expanding the candidate set $\mathcal{A}_{\boldsymbol{x}_{<t}}$ depends on the rationality of the token. This rationality is determined by the extent to which the model's confidence increases when considering the inclusion of this token.
• Figure 4: Unit tests of the truncation behavior of top-$p$, typical, $\eta$-sampling and adaptive decoding on CheckList-inspired prefixes. $p=0.90$, typical$=0.90$, $\eta=\sqrt{0.0009}=0.03$, and threshold for adaptive decoding $\epsilon$ is $0.0005$.
• Figure 5: Human written text: "Barack Obama was born in 1961. He was raised in Hawaii by his mother and grandparents. Growing up, Obama faced the challenges of being biracial, with a Kenyan father and an American mother. Despite these challenges, he excelled academically and eventually attended Columbia University in New York City."

Theorems & Definitions (9)

• Definition 3.1
• Theorem 3.2
• Definition 3.3
• Corollary 3.4
• Corollary 3.1
• proof
• Corollary 3.2
• Corollary 3.3
• proof