Bayesian Evaluation of Large Language Model Behavior

TL;DR

The paper tackles the problem of evaluating LLM-based systems under stochastic text generation by quantifying uncertainty in binary behavioral metrics through a Bayesian Beta-Binomial per-prompt model. It defines per-prompt success probabilities $\theta_m$ and an aggregate metric $W=g(\theta_1,...,\theta_M)$, and demonstrates how to obtain posterior distributions for these quantities as data are collected. Two case studies—pairwise preferences between LLMs and refusals of harmful inputs—illustrate how Bayesian uncertainty quantification provides richer insight than deterministic, greedy evaluations, including credible intervals and per-prompt uncertainty. The authors extend the framework to sequential evaluation via multi-armed bandits (greedy and Thompson sampling), enabling cost-efficient data collection that rapidly reduces uncertainty, with applications to safety auditing and model comparisons in practice.

Abstract

It is increasingly important to evaluate how text generation systems based on large language models (LLMs) behave, such as their tendency to produce harmful output or their sensitivity to adversarial inputs. Such evaluations often rely on a curated benchmark set of input prompts provided to the LLM, where the output for each prompt may be assessed in a binary fashion (e.g., harmful/non-harmful or does not leak/leaks sensitive information), and the aggregation of binary scores is used to evaluate the LLM. However, existing approaches to evaluation often neglect statistical uncertainty quantification. With an applied statistics audience in mind, we provide background on LLM text generation and evaluation, and then describe a Bayesian approach for quantifying uncertainty in binary evaluation metrics. We focus in particular on uncertainty that is induced by the probabilistic text generation strategies typically deployed in LLM-based systems. We present two case studies applying this approach: 1) evaluating refusal rates on a benchmark of adversarial inputs designed to elicit harmful responses, and 2) evaluating pairwise preferences of one LLM over another on a benchmark of open-ended interactive dialogue examples. We demonstrate how the Bayesian approach can provide useful uncertainty quantification about the behavior of LLM-based systems.

Figures (15)

• Figure 1: Estimated distribution of $W_{mean}$ after $n=10$ (left) and $n=50$ (right) stochastic text generations using 10,000 Monte Carlo draws.
• Figure 2: Poisson binomial probability mass function of $W_{>\nu}$ for $\nu=0.75$ after observing $n=10$ (left) and $n=50$ (right) stochastic text generations. The dotted gray line indicates that Model A was preferred on 41/80 inputs using greedy decoding.
• Figure 3: Poisson binomial probability mass function of $W_{>\nu}$ for $\nu=0.95$ after observing $n=10$ (left) and $n=50$ (right) stochastic text generations. The dotted gray line indicates that 98/100 inputs were refused when using a greedy decoding strategy.
• Figure 4: Estimated distribution of $W_{\min}$ for $n=10$ (left) and $n=50$ (right) stochastic text generations using 10,000 Monte Carlo draws.
• Figure 5: Results using the sequential algorithms on MT-Bench for $W_{>\nu}$ with $\nu = 0.75$, $M=80$, averaged over 1000 runs. Shaded regions indicate interquartile ranges (25th-75th percentiles).