A Contrastive Framework for Neural Text Generation

TL;DR

This work identifies degeneration in neural text generation as an outcome of anisotropic token representations and introduces a contrastive framework (SimCTG) to learn isotropic, discriminative token spaces. It pairs this training with contrastive search decoding, which selects high-probability candidates while penalizing degeneration to preserve coherence. Across document generation and open-domain dialogue tasks in multiple languages, the approach yields significant improvements in perplexity, token coherence, diversity, and human judgments, outperforming strong baselines and even approaching human fluency in some settings. The analyses further reveal that isotropy in representations, margin tuning, and decoding dynamics jointly govern performance, suggesting broad applicability to future large-scale transformers and constrained/dependent generation tasks.

Abstract

Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.

Figures (8)

• Figure 1: Token cosine similarity matrix of (a) GPT-2 and (b) SimCTG. (best viewed in color)
• Figure 2: Layer-wise representation self-similarity.
• Figure 3: The effect of contrastive margin $\rho$.
• Figure 4: Contrastive search vs nucleus sampling.
• Figure 5: Inference latency comparison.