Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reranking Laws for Language Generation: A Communication-Theoretic Perspective

António Farinhas, Haau-Sing Li, André F. T. Martins

TL;DR

This work reframes generator–reranker LLM systems as a communication channel with a sender (the generator) transmitting $N$ descriptions through parallel, noisy channels and a receiver (the reranker) decoding by ranking. It derives conditions under which the overall system achieves asymptotically vanishing error $P_{\mathrm{err}}(N;q)\to0$ as $N\to\infty$, covering independent hypotheses with perfect, Mallows, and Zipf–Mandelbrot rerankers, as well as dependent (exchangeable) hypotheses with a Beta prior on the per-hypothesis error. Theoretical results yield explicit decay rates (e.g., exponential for Mallows, power-law under Beta coupling) and confirm robustness to certain dependencies; these are validated empirically on text-to-code generation and medical machine translation, with fits that capture the observed improvement in failure rates as $N$ grows. The findings suggest practical reranking laws for predicting how many hypotheses are needed to meet a target reliability and point to future directions involving interactive feedback and external verifiers to further improve LLM safety and reliability.

Abstract

To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free (i.e., yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.

Reranking Laws for Language Generation: A Communication-Theoretic Perspective

TL;DR

This work reframes generator–reranker LLM systems as a communication channel with a sender (the generator) transmitting descriptions through parallel, noisy channels and a receiver (the reranker) decoding by ranking. It derives conditions under which the overall system achieves asymptotically vanishing error as , covering independent hypotheses with perfect, Mallows, and Zipf–Mandelbrot rerankers, as well as dependent (exchangeable) hypotheses with a Beta prior on the per-hypothesis error. Theoretical results yield explicit decay rates (e.g., exponential for Mallows, power-law under Beta coupling) and confirm robustness to certain dependencies; these are validated empirically on text-to-code generation and medical machine translation, with fits that capture the observed improvement in failure rates as grows. The findings suggest practical reranking laws for predicting how many hypotheses are needed to meet a target reliability and point to future directions involving interactive feedback and external verifiers to further improve LLM safety and reliability.

Abstract

To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free (i.e., yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.
Paper Structure (34 sections, 4 theorems, 24 equations, 8 figures)

This paper contains 34 sections, 4 theorems, 24 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Notation.
  3. A Communication-Theoretic Perspective of Generator-Reranker Systems
  4. Generator-Reranker Systems with Independent Hypotheses
  5. Perfect and random rerankers
  6. Imperfect reranker: Mallows model
  7. Imperfect reranker: Zipf-Mandelbrot model
  8. Generator-Reranker Systems with Dependent Hypotheses
  9. Perfect reranker and Beta coupling.
  10. Imperfect reranker.
  11. Experiments
  12. Code generation
  13. Machine translation
  14. Discussion and Related Work
  15. Conclusions
  16. ...and 19 more sections

Key Result

proposition 1

When $R$ is a Mallows reranker, for any $\lambda > 0$, the protocol is asymptotically error-free and the error probability decays exponentially fast, $P_\mathrm{err}(N; q) = \mathcal{O}((e^{-\lambda}(1-\epsilon) + \epsilon)^N)$.

Figures (8)

  • Figure 1: Left: A generator-reranker system $(G, R)$ depicted as a communication system (\ref{['sec:A Communication-Theoretic View of Reranking']}). Given a query $q$ with acceptance set $\mathcal{X}(q)$, the sender sends $N$ descriptions through noisy channels. The receiver's goal is to decode an acceptable answer through reranking. Right: Graphical model of the generator $G$. We consider two different models: a simplified version with $N$ independent hypotheses, represented in black (\ref{['sec:Independent hypotheses']}), and a scenario with exchangeable hypotheses, represented in red (\ref{['sec:Exchangeable hypotheses']}).
  • Figure 2: Log of the failure rate (difference with respect to the baseline rate $\log \epsilon$) as a function of the number of generated independent hypotheses $N$ for several values of $e^{-\lambda}$ and $\epsilon=0.3$. Left: Mallows model (\ref{['subsec:mallows']}). Right: Zipf-Mandelbrot model (\ref{['subsec:zipf']}).
  • Figure 3: Log of the failure rate as a function of the number of generated exchangeable hypotheses $N$ for several values of $\gamma$, $e^{-\lambda}$, and $\epsilon=\alpha=0.3$.
  • Figure 4: Log of the failure rate as a function of $N$. The empirical data is represented with dots (left: dev, right: test set) and our fitted models with solid and dashed lines (imperfect and perfect reranker, respectively). Top: text-to-code generation (\ref{['subsec:Code generation']}). Bottom: machine translation (\ref{['subsec:Machine translation']}).
  • Figure 5: $P_\mathrm{err}$ using rerankers with probability mass function $\eta_j \propto (N-j+1)^r$ with $r=\{1,2,3\}$ (from left to right) and $\epsilon=0.8$. The resulting protocol is not asymptotically error-free: the horizontal asymptotes in red correspond to $\epsilon^{r+1}$, according to \ref{['eq:asymptote']}.
  • ...and 3 more figures

Theorems & Definitions (5)

  • definition 1
  • proposition 1
  • proposition 2
  • proposition 3
  • proposition 4