Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation

Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, Matthias Aßenmacher

Abstract

Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, $k-$sampling, nucleus $p-$sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available.

Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation

Abstract

Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, sampling, nucleus sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available.
Paper Structure (37 sections, 1 theorem, 16 equations, 10 figures, 9 tables)

This paper contains 37 sections, 1 theorem, 16 equations, 10 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Deterministic Methods.
  5. Stochastic Methods.
  6. Methodology
  7. Incorporating Model Uncertainty
  8. Theoretical Motivation
  9. Experimental Setup
  10. Evaluation Metrics
  11. Diversity.
  12. MAUVE.
  13. Coherence.
  14. Human Eval.
  15. Datasets
  16. ...and 22 more sections

Key Result

Proposition 1

The degeneration penalty $\max_j \{s(h_v, h_{x_j})\}$ is a function of the penalty $||h_v - h_{x_j} ||_2$ in statistical Tikhonov-regularization, if the representations $h$ are normalized.

Figures (10)

  • Figure 1: Visualization of the Adaptive Contrastive Search (ACS) process: A three-step procedure that uses entropy as a proxy for model uncertainty to automatically adjust contrastive search parameters.
  • Figure 2: Visualization of uncertainty over time, measured by the Shannon entropy of the output distribution (first row, left). It is used to determine the value of $k$ over time (right). The second row illustrates the entropy of the top-$k$ tokens distribution, which is used to compute the value of $\alpha$ (right).
  • Figure 3: Decoding behavior over time from a Wikinews prompt, $q = 1$.
  • Figure 4: Decoding behavior over time from a Wikinews prompt, $q = 8$.
  • Figure 5: Decoding behavior over time from a Wikitext prompt, $q = 1$.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Proposition 1
  • Proof 1