Estimating the Error of Large Language Models at Pairwise Text Comparison

TL;DR

The model outperforms the biased Bradley-Terry model and the commutativity score in indicating LLMs'error at this task and obtains consistent estimates of LLMs'error.

Abstract

We measure LLMs' output error at pairwise text comparison, noting the probability of error in their preferences. Our method does not rely on the ground truth and supports two scenarios: (i) uniform error rate regardless of the order of comparison, estimated with two comparisons for each text pair with either text placed first; (ii) binary positional bias assuming distinct error rates for the two orders of comparison, estimated with repeated comparisons between the texts. The Copeland counting constructs a ranking over the compared texts from pairwise preferences; the ranking reveals the poor scalability of LLM-based pairwise comparison and helps yield the estimates for LLMs' error rates. We apply the method to six LLMs (ChatGPT, Claude, DeepSeek, Gemini, Grok, Qwen) with five types of text input and obtain consistent estimates of LLMs' error. In general, the measured two positional bias terms are similar, close to the uniform error. Considering both the error rates and the robustness to the variation of prompts, Claude obtained the most desirable performance in this experiment. Our model outperforms the biased Bradley-Terry model and the commutativity score in indicating LLMs' error at this task.

Figures (8)

• Figure 1: Scalability of the Copeland ranking based on pairwise comparisons. Showing the probability of the $\#m$-ranking text obtaining the correct Copeland score, $P(R_{m},TO)$. $m = 1$ to 8. (a) Uniform error. $S_{Copeland}^{Z}$ calculated on $\mathbf{Z}$. $\epsilon = 0.1$ to 0.5. Inset: $\epsilon = 0.5$. (b1)-(b4) Positional bias. $S^{W}_{Copeland}$ calculated on $\mathbf{W}$. Values of $\epsilon_+, \epsilon_-$ and $k_+, k_-$ are shown on the plots.
• Figure 2: Estimation of LLMs' error rate from the output preferences (uniform error $\epsilon$). Input text: short poems. LLM: ChatGPT. (a) Observation matrix $\mathbf{Z}$ constructed from pairwise preferences $\hat{y}_{ij}$. Red/Blue/White: -1/0/1 entries. (b) Copeland scores $S_{Copeland}^Z$ calculated on $\mathbf{Z}$. Highest/lowest scores (98/-95) are obtained at the corresponding columns of $\mathbf{Z}$. Dash line: perfect scores. The area between the dash line and the observed $S_{Copeland}^Z$ is the difference $\Delta_{S,obs}$. (c) Fitting the $\Delta_{S,obs}-N$ curve. Light curves: synthetic $\Delta_{S, \epsilon}-N$ calculated with $\epsilon\in[0.05,0.5]$ at interval 0.05. Thick curves: the empirical $\Delta_{S,obs}-N$. Inset: grid search ($\epsilon\in[0,0.5]$ at interval 0.005) minimizing the summed distance between the empirical and the synthetic curves. Best-fit $\epsilon = 0.130$, i.e., a uniform error rate of 13.0$\%$. Results from three empirical runs of comparison largely overlap (different colors at the thick curves and in the inset), which have the commutativity scores $S_{com} = 39.01\%, 38.95\%, 39.03\%$, respectively; three instances of estimate are identical.
• Figure 3: Estimation of LLMs' error rate from the output preferences (positional bias $\epsilon_{+/-}$). Input text: short poems. LLM: ChatGPT. Minimizing the summed distance between the empirical $\Delta_{S,obs}-N$ and the synthetic $\Delta_{S}-N$ curves. 2-D grid search $\epsilon_+,\epsilon_-\in[0,0.5]$ at interval 0.005. The main heatmap shows the result averaging the nine cases of $\mathbf{W}^{1,1}$ to $\mathbf{W}^{3,3}$ (left inset). Diagonal elements (right inset) concur with the curve at uniform error. Best-fit $\epsilon_+ = 0.155, \epsilon_- = 0.100$ on the average $\mathbf{W}$.
• Figure 4: Main estimation results. Grid search for $\epsilon, \epsilon_{+/-}\in[0,0.5]$ at interval 0.005. Crosses: $\epsilon$ estimates at uniform error (using matrix $\mathbf{Z}$). Other markers: $\epsilon_{+/-}$ estimates at positional bias (using matrix $\mathbf{W}$). (a)-(e): Five types of text input. (f) Average estimates: over the five types (main), over pseudo-word and pseudo paragraphs (upper inset), over the other three types (lower inset).
• Figure 5: Estimation with the biased Bradley-Terry model. Estimation of $\epsilon$ with $\mathbf{X}^{3,3}$. Input: short poems. LLM: ChatGPT. (a) Convergence of iterations. Initial scores $\hat{s}_{1:N}\in uniform\ [0,1]$. Y-axis: number of texts with unchanged ranking across one iteration. X-axis: number of iterations. (b) Grid search ($\epsilon$ at interval 0.005) minimizing the spread of estimated scores $\hat{s}$. Minimum spread obtained at $\epsilon_{opt}=0.05$. Inset: frequency of $\epsilon_{opt}$ across 200 random seeds for the initial scores (resolution 0.02, i.e., 25 bins within [0, 0.5]).