Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perplexity Cannot Always Tell Right from Wrong

Petar Veličković, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, Razvan Pascanu

TL;DR

Perplexity, a common proxy for model quality, can be misleading for model selection, especially with long contexts and distribution shifts. Grounded in a continuity theorem for decoder-only Transformers, the paper proves that high-confidence copying of long sequences implies the existence of inputs the model gets wrong yet with log-perplexity approaching zero. It introduces iso-perplexity analysis and demonstrates, via parity and bitstring copy experiments, that perplexity can fail to prefer the more accurate model, particularly under OOD conditions. The work advocates diagnostic approaches and confidence-aware metrics to complement perplexity for safer, more reliable model selection and deployment.

Abstract

Perplexity -- a function measuring a model's overall level of "surprise" when encountering a particular output -- has gained significant traction in recent years, both as a loss function and as a simple-to-compute metric of model quality. Prior studies have pointed out several limitations of perplexity, often from an empirical manner. Here we leverage recent results on Transformer continuity to show in a rigorous manner how perplexity may be an unsuitable metric for model selection. Specifically, we prove that, if there is any sequence that a compact decoder-only Transformer model predicts accurately and confidently -- a necessary pre-requisite for strong generalisation -- it must imply existence of another sequence with very low perplexity, but not predicted correctly by that same model. Further, by analytically studying iso-perplexity plots, we find that perplexity will not always select for the more accurate model -- rather, any increase in model confidence must be accompanied by a commensurate rise in accuracy for the new model to be selected.

Perplexity Cannot Always Tell Right from Wrong

TL;DR

Perplexity, a common proxy for model quality, can be misleading for model selection, especially with long contexts and distribution shifts. Grounded in a continuity theorem for decoder-only Transformers, the paper proves that high-confidence copying of long sequences implies the existence of inputs the model gets wrong yet with log-perplexity approaching zero. It introduces iso-perplexity analysis and demonstrates, via parity and bitstring copy experiments, that perplexity can fail to prefer the more accurate model, particularly under OOD conditions. The work advocates diagnostic approaches and confidence-aware metrics to complement perplexity for safer, more reliable model selection and deployment.

Abstract

Perplexity -- a function measuring a model's overall level of "surprise" when encountering a particular output -- has gained significant traction in recent years, both as a loss function and as a simple-to-compute metric of model quality. Prior studies have pointed out several limitations of perplexity, often from an empirical manner. Here we leverage recent results on Transformer continuity to show in a rigorous manner how perplexity may be an unsuitable metric for model selection. Specifically, we prove that, if there is any sequence that a compact decoder-only Transformer model predicts accurately and confidently -- a necessary pre-requisite for strong generalisation -- it must imply existence of another sequence with very low perplexity, but not predicted correctly by that same model. Further, by analytically studying iso-perplexity plots, we find that perplexity will not always select for the more accurate model -- rather, any increase in model confidence must be accompanied by a commensurate rise in accuracy for the new model to be selected.
Paper Structure (25 sections, 7 theorems, 25 equations, 4 figures)

This paper contains 25 sections, 7 theorems, 25 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Failures of Perplexity in Long-Context
  4. Confidence and Calibration
  5. Theoretical Results on Transformers
  6. Log-perplexity of wrong next-token predictors can arbitrarily approach zero
  7. Preliminaries
  8. Deterministic sampling
  9. Stochastic sampling
  10. Implications on learning dynamics
  11. Empirical analysis
  12. An analytic view into confidence
  13. Preliminaries
  14. Iso-perplexity curves
  15. Iso-perplexity at $a=0.5$
  16. ...and 10 more sections

Key Result

Lemma 3.1

Let $T$ be a decoder-only Transformer with compact position embeddings (CPE), as defined by pasten2025continuity. Assume $T$ is trained to perform a copy task over bitstrings, and it samples outputs by greedy decoding. Let $\boldsymbol\alpha = \alpha_1\alpha_2\cdots\alpha_n\cdots$ be an infinite bit Then, for every $\xi > 0$, there must exist $n'\in\mathbb{N}$ such that, for all prefixes $\boldsym

Figures (4)

  • Figure 1: Using the continuity result of pasten2025continuity, we show that, if a (compact) Transformer $T$ is confident in copying any long enough sequence $\boldsymbol\alpha_N$, then there must exist $\boldsymbol\beta_N$ which $T$ fails to copy, yet, log-perplexity will tend to zero as $N$ grows.
  • Figure 2: For various sequence lengths, $N$, on the copy task, we compute ( Left) the $L_\infty$ norm of the difference between the logit distributions across all positions, ( Middle) the minimal observed probability of predicting $\boldsymbol{\alpha}_k$---our conservative estimate of $1-\gamma$---and the maximal observed probability of predicting $\boldsymbol{\beta}_N$---which can serve as a bound on the probability that the model will copy $\boldsymbol{\beta}_N$ properly. We also plot ( Right) the log-perplexity for both $\boldsymbol\alpha_N$ and $\boldsymbol{\beta}_N$. This is done both for ( Top) a toy copy environment where a CPE Transformer is trained on sizes up to 16 bits, and ( Bottom:) prompting Gemma 3 4B with a copy request.
  • Figure 3: Left: Iso-perplexity curves for the setting with an unreliable base model ($a=0.5$) for varying choices of confidence ($1-\gamma$). Right: Iso-perplexity curves for an unconfident base model ($\gamma=0.4$) for varying choice of base accuracy $a$,.
  • Figure 4: Scatter plots of micro-$\mathrm{F}_1$ scores against log-perplexities, $L$, for various checkpoints of a Transformer model trained on the Parity problem, as specified by vitvitskyi2025makes, for both in-distribution ( Left) and out-of-distribution ( Right) held-out data. We also colour-code the checkpoints by their averaged Shannon entropy, $\bar{H}$, provide the Pearson correlation coefficient, $r$, and highlight the point with the highest accuracy by using a star (also colour-coded by entropy).

Theorems & Definitions (13)

  • Lemma 3.1: Perplexity convergence
  • Proposition 3.2: Collapsing confidence
  • proof
  • Corollary 3.3
  • Remark 3.4
  • Remark 3.5
  • Proposition 3.6
  • proof : Informal proof
  • Corollary 3.7: Vanishing gradients on incorrect samples
  • Lemma 1.1: 3.1.: Perplexity convergence
  • ...and 3 more