Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction

Eitan Wagner, Omri Abend

TL;DR

The paper distinguishes three distinct LM output interpretations: source-distribution (distribution estimation), target/world-event distribution (target distribution estimation), and response prediction. It formalizes these concepts and traces how pretraining, instruction tuning, and RLHF shape the resulting probabilities, arguing that many studies implicitly assume their equivalence, leading to misinterpretations. It analyzes two main inference strategies—logit probabilities and explicit probability reports—and maps them to text completion, response generation, and event modeling use cases, showing that only explicit probability reporting reliably furnishes unbiased event probabilities. The work provides a rigorous framework to interpret LM outputs, cautions against conflating completion, response, and event distributions, and highlights implications for world modeling and alignment research.

Abstract

The notion of language modeling has gradually shifted in recent years from a distribution over finite-length strings to general-purpose prediction models for textual inputs and outputs, following appropriate alignment phases. This paper analyzes the distinction between distribution estimation and response prediction in the context of LLMs, and their often conflicting goals. We examine the training phases of LLMs, which include pretraining, in-context learning, and preference tuning, and also the common use cases for their output probabilities, which include completion probabilities and explicit probabilities as output. We argue that the different settings lead to three distinct intended output distributions. We demonstrate that NLP works often assume that these distributions should be similar, which leads to misinterpretations of their experimental findings. Our work sets firmer formal foundations for the interpretation of LLMs, which will inform ongoing work on the interpretation and use of LLMs' induced distributions.

What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction

TL;DR

The paper distinguishes three distinct LM output interpretations: source-distribution (distribution estimation), target/world-event distribution (target distribution estimation), and response prediction. It formalizes these concepts and traces how pretraining, instruction tuning, and RLHF shape the resulting probabilities, arguing that many studies implicitly assume their equivalence, leading to misinterpretations. It analyzes two main inference strategies—logit probabilities and explicit probability reports—and maps them to text completion, response generation, and event modeling use cases, showing that only explicit probability reporting reliably furnishes unbiased event probabilities. The work provides a rigorous framework to interpret LM outputs, cautions against conflating completion, response, and event distributions, and highlights implications for world modeling and alignment research.

Abstract

The notion of language modeling has gradually shifted in recent years from a distribution over finite-length strings to general-purpose prediction models for textual inputs and outputs, following appropriate alignment phases. This paper analyzes the distinction between distribution estimation and response prediction in the context of LLMs, and their often conflicting goals. We examine the training phases of LLMs, which include pretraining, in-context learning, and preference tuning, and also the common use cases for their output probabilities, which include completion probabilities and explicit probabilities as output. We argue that the different settings lead to three distinct intended output distributions. We demonstrate that NLP works often assume that these distributions should be similar, which leads to misinterpretations of their experimental findings. Our work sets firmer formal foundations for the interpretation of LLMs, which will inform ongoing work on the interpretation and use of LLMs' induced distributions.
Paper Structure (47 sections, 5 equations, 4 figures)

This paper contains 47 sections, 5 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background: Distribution Estimation and Response Prediction
  3. Distribution Estimation
  4. Target distribution estimation.
  5. Response Prediction
  6. Praxis: LM Training and Inference
  7. Probabilistic Language Modeling
  8. Training Stages
  9. T1. Pretraining.
  10. T2. Adaptation/Fine-tuning:
  11. Inference
  12. I1. Logit probabilities:
  13. I2. Explicit probability report:
  14. Use Cases for the Different Strategies
  15. Text completion
  16. ...and 32 more sections

Figures (4)

  • Figure 1: An example where distribution estimation (for language), target distribution estimation (for events), and response prediction, lead to different output probabilities. Models commonly have a strong bias to generate Heads, and the generated utterances will yield a probability with this additional bias (Case 1a). The event probabilities can be obtained by explicitly reporting them (Case 1b). Utterances that rationally predict an event (Case 2) should always choose the mode.
  • Figure 2: Overview of training methods.
  • Figure 3: Overview of inference methods.
  • Figure 4: Overview of probability interpretations and their mapping to task type (\ref{['sec:estimation_prediction']}), training strategy (\ref{['sec:training']}), inference strategy (\ref{['sec:inference']}), and data-generation (\ref{['sec: analysis']}).