Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse

Alkis Kalavasis, Anay Mehrotra, Grigoris Velegkas

TL;DR

This work formalizes the trade-off between hallucination (hallucinating invalid outputs) and mode collapse (limited breadth) in language generation within a Gold–Angluin inspired probabilistic framework. It introduces the generation error and breadth notions, and studies them for countable language collections under a membership oracle (MOP). The key results show a fundamental tension: for broad, practically relevant iterative generators (including next-token models), generation with breadth is impossible at any rate if the language collection is not identifiable in the limit; however, generation with breadth becomes achievable when negative examples are available, and universal exponential rates are possible for generation without breadth. The paper further analyzes relaxations like unambiguous generation and approximate breadth, proving strong impossibility results under stability and decidable MOP, while providing conditions under which exponential rates can be obtained using subset or positive/negative feedback, highlighting the potential of post-training feedback to mitigate hallucinations and reduce mode collapse in real-world systems.

Abstract

Specifying all desirable properties of a language model is challenging, but certain requirements seem essential. Given samples from an unknown language, the trained model should produce valid strings not seen in training and be expressive enough to capture the language's full richness. Otherwise, outputting invalid strings constitutes "hallucination," and failing to capture the full range leads to "mode collapse." We ask if a language model can meet both requirements. We investigate this within a statistical language generation setting building on Gold and Angluin. Here, the model receives random samples from a distribution over an unknown language K, which belongs to a possibly infinite collection of languages. The goal is to generate unseen strings from K. We say the model generates from K with consistency and breadth if, as training size increases, its output converges to all unseen strings in K. Kleinberg and Mullainathan [KM24] asked if consistency and breadth in language generation are possible. We answer this negatively: for a large class of language models, including next-token prediction models, this is impossible for most collections of candidate languages. This contrasts with [KM24]'s result, showing consistent generation without breadth is possible for any countable collection of languages. Our finding highlights that generation with breadth fundamentally differs from generation without breadth. As a byproduct, we establish near-tight bounds on the number of samples needed for generation with or without breadth. Finally, our results offer hope: consistent generation with breadth is achievable for any countable collection of languages when negative examples (strings outside K) are available alongside positive ones. This suggests that post-training feedback, which encodes negative examples, can be crucial in reducing hallucinations while limiting mode collapse.

On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse

TL;DR

This work formalizes the trade-off between hallucination (hallucinating invalid outputs) and mode collapse (limited breadth) in language generation within a Gold–Angluin inspired probabilistic framework. It introduces the generation error and breadth notions, and studies them for countable language collections under a membership oracle (MOP). The key results show a fundamental tension: for broad, practically relevant iterative generators (including next-token models), generation with breadth is impossible at any rate if the language collection is not identifiable in the limit; however, generation with breadth becomes achievable when negative examples are available, and universal exponential rates are possible for generation without breadth. The paper further analyzes relaxations like unambiguous generation and approximate breadth, proving strong impossibility results under stability and decidable MOP, while providing conditions under which exponential rates can be obtained using subset or positive/negative feedback, highlighting the potential of post-training feedback to mitigate hallucinations and reduce mode collapse in real-world systems.

Abstract

Specifying all desirable properties of a language model is challenging, but certain requirements seem essential. Given samples from an unknown language, the trained model should produce valid strings not seen in training and be expressive enough to capture the language's full richness. Otherwise, outputting invalid strings constitutes "hallucination," and failing to capture the full range leads to "mode collapse." We ask if a language model can meet both requirements. We investigate this within a statistical language generation setting building on Gold and Angluin. Here, the model receives random samples from a distribution over an unknown language K, which belongs to a possibly infinite collection of languages. The goal is to generate unseen strings from K. We say the model generates from K with consistency and breadth if, as training size increases, its output converges to all unseen strings in K. Kleinberg and Mullainathan [KM24] asked if consistency and breadth in language generation are possible. We answer this negatively: for a large class of language models, including next-token prediction models, this is impossible for most collections of candidate languages. This contrasts with [KM24]'s result, showing consistent generation without breadth is possible for any countable collection of languages. Our finding highlights that generation with breadth fundamentally differs from generation without breadth. As a byproduct, we establish near-tight bounds on the number of samples needed for generation with or without breadth. Finally, our results offer hope: consistent generation with breadth is achievable for any countable collection of languages when negative examples (strings outside K) are available alongside positive ones. This suggests that post-training feedback, which encodes negative examples, can be crucial in reducing hallucinations while limiting mode collapse.

Paper Structure

This paper contains 83 sections, 54 theorems, 133 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Informal Results
  3. Setup and Definitions
  4. Main Results
  5. A Family of Generators for which $\mathsf{MOP}{}(\cdot)$ Is Decidable.
  6. Implications for the Gold--Angluin Model.
  7. Results with Two Relaxations of Breadth.
  8. Organization of Rest of the Introduction.
  9. Technical Overview
  10. A Natural Strategy to Prove \ref{['infthm:breadth:mop']}.
  11. Initial Attempts to Implement the Strategy.
  12. Our Approach.
  13. Additional Results with Relaxation of Consistency and Breadth
  14. Relaxation 1: Unambiguous Generation
  15. Relaxation 2: Consistency with Approximate Breadth
  16. ...and 68 more sections

Key Result

Theorem 2.1

Let $\euscr{L} = \{L_\infty, L_1, L_2, \dots\}$ be the language collection with $L_1 \subset L_2 \subset \dots \subset L_\infty = \cup_{i \geq 1} L_i$ and for each $i,$$\left| L_i \right| < \infty$. Then, there is no algorithm that identifies $\euscr{L}$ in the limit from positive examples. Moreover

Figures (6)

  • Figure 1: An Unambiguous Generator That neither Has Consistency nor Breadth. In this example, the language collection $\euscr{L}$ has two languages $L$ and $K$, where $K$ denotes the target language. The red curve denotes $L$, the dashed green curve denotes $K$, and the blue curve denotes the support of $\operatorname{supp}(\mathpzc{G}_n).$ The generator $\mathpzc{G}_n$ hallucinates since $\operatorname{supp}(\mathpzc{G}_n) \setminus K \neq \emptyset$ and does not achieve breadth for the target $K$ since $B=K\setminus \operatorname{supp}(\mathpzc{G}_n)$ is non-empty. Nevertheless, this generator is unambiguous as $\left| \operatorname{supp}(\mathpzc{G}_n) \setminus K \right| + \left| B \right| < \left| \operatorname{supp}(\mathpzc{G}_n) \setminus L \right| + \left| A \right|.$
  • Figure 2: Outline of Proof of \ref{['thm:dichotomy-identification-positive']}
  • Figure 3: Outline of Proof of \ref{['thm:statistical-generation']}
  • Figure 4: Illustrations of language collections that are (a,b) trivial for generation and (c) non-trivial for generation. In cases (a) and (c), the collection $\euscr{L}$ has three languages -- $L_1,L_2,$ and $L_3$ -- denoted by different colors. Case (b) illustrates \ref{['ex:prefixes-of-rationals']}; here, the collection $\euscr{L}$ has infinitely many languages which follow a nested structure $L_1\supsetneq L_2\supsetneq \dots \supsetneq \left\{0\right\}$.
  • Figure 5: Outline of Proof of \ref{['mainthm:gen:mop']}
  • ...and 1 more figures

Theorems & Definitions (126)

  • Definition 1: Valid Distribution angluin1988identifying
  • Definition 2: Consistency
  • Definition 3: Informal, Universal Rates; bousquet2021theory, see \ref{['def:achievable-rates']}
  • Definition 4: Breadth
  • Definition 5: Membership Oracle Problem
  • Definition 6: MOP for Generating Algorithms
  • Definition 7: Stability
  • Definition 8: Unambiguous Generator
  • Definition 9: Generation with Approximate Breadth
  • Definition 10: Language Identification in the Limit gold1967language
  • ...and 116 more