Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores

Zachary Robertson

TL;DR

This work shows that TVD-MI based evaluation of LLMs yields additive score structure on the raw $[-1,1]$ scale when averaged over peer comparisons, and that preserving this geometry with an identity link significantly outperforms traditional logistic or probit mappings. By deriving a clipped-linear model from Gini entropy and enforcing box constraints, the authors achieve accurate, sparse recovery of agent–item scores $s_{ij} = \theta_i - b_j$ with strong ranking fidelity (e.g., $\rho \approx 0.97$) using only $\sim 33\%$ of full evaluations. The approach maintains robustness across multiple domains (PubMed, OPUS, ICLR) and judges, supports tractable optimization, and provides practical recommendations for efficient, ground-truth-free LLM evaluation in bounded-response settings. The results suggest broad applicability to any bounded-response domain where scores arise from averaging and emphasize the value of validating additivity via a discrete integrability test before adopting sparse evaluation.

Abstract

Pairwise comparisons of large language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions. Maximum-likelihood approaches to IRT use logistic links, but we find empirically that these transformations introduce curvature that breaks additivity: across three domains, the identity link yields median curl on raw data of 0.080-0.150 (P95 = [0.474, 0.580]), whereas probit/logit introduce substantially higher violations (median [0.245, 0.588], P95 [0.825, 2.252]). We derive this clipped-linear model from Gini entropy maximization, yielding a box-constrained least-squares formulation that handles boundary saturation. At 33% coverage, we achieve holdout RMSE $0.117 \pm 0.008$ while preserving agent rankings (Spearman $ρ= 0.972 \pm 0.015$), three times fewer evaluations than full dense. Judge robustness analysis (GPT-4o-mini vs. Llama3-70b) shows strong agreement in agent rankings ($ρ= 0.872$) and consistent identity-link advantage. TVD-MI's geometry is best preserved by identity mapping for efficient LLM evaluation, applicable to other bounded-response domains.

Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores

TL;DR

This work shows that TVD-MI based evaluation of LLMs yields additive score structure on the raw scale when averaged over peer comparisons, and that preserving this geometry with an identity link significantly outperforms traditional logistic or probit mappings. By deriving a clipped-linear model from Gini entropy and enforcing box constraints, the authors achieve accurate, sparse recovery of agent–item scores with strong ranking fidelity (e.g., ) using only of full evaluations. The approach maintains robustness across multiple domains (PubMed, OPUS, ICLR) and judges, supports tractable optimization, and provides practical recommendations for efficient, ground-truth-free LLM evaluation in bounded-response settings. The results suggest broad applicability to any bounded-response domain where scores arise from averaging and emphasize the value of validating additivity via a discrete integrability test before adopting sparse evaluation.

Abstract

Pairwise comparisons of large language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions. Maximum-likelihood approaches to IRT use logistic links, but we find empirically that these transformations introduce curvature that breaks additivity: across three domains, the identity link yields median curl on raw data of 0.080-0.150 (P95 = [0.474, 0.580]), whereas probit/logit introduce substantially higher violations (median [0.245, 0.588], P95 [0.825, 2.252]). We derive this clipped-linear model from Gini entropy maximization, yielding a box-constrained least-squares formulation that handles boundary saturation. At 33% coverage, we achieve holdout RMSE while preserving agent rankings (Spearman ), three times fewer evaluations than full dense. Judge robustness analysis (GPT-4o-mini vs. Llama3-70b) shows strong agreement in agent rankings () and consistent identity-link advantage. TVD-MI's geometry is best preserved by identity mapping for efficient LLM evaluation, applicable to other bounded-response domains.
Paper Structure (48 sections, 4 theorems, 27 equations, 2 figures, 6 tables)

This paper contains 48 sections, 4 theorems, 27 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Background and Related Work
  4. Item Response Theory and Matrix Completion
  5. Peer Prediction and TVD-MI
  6. Why Traditional IRT Links Fail for TVD-MI
  7. Experimental Setup
  8. Model and Diagnostics
  9. Clipped-Linear Model from Gini Entropy
  10. Discrete Integrability Test
  11. Link and Model Ablation
  12. Connectivity Requirements
  13. Sampling and Protocol
  14. Results
  15. Sample Efficiency
  16. ...and 33 more sections

Key Result

Lemma 2.1

Total variation admits the IPM representation and the supremum is achieved by a binary critic $h^*(x)\in\{-1,1\}$ with $h^*(x)=\mathrm{sign}(p(x)-q(x))$ almost everywhere. Moreover, among $f$-divergences, total variation is (up to a positive multiplicative constant) the only one that is also an IPM over a symmetric convex function class (the $L

Figures (2)

  • Figure 1: Identity link preserves natural additivity of TVD-MI scores. Empirical cumulative distribution functions of curl magnitude $|\Delta|$ on raw score matrices across three domains (20,000 paired rectangles per condition). Identity is leftmost with median curl 0.080–0.150, while curved links shift right with 2–7× higher violations (logit median: 0.438–0.588). TVD-MI averaging produces inherently additive data in $[-1,1]$ space; nonlinear links warp this geometry.
  • Figure 2: Hold-out RMSE vs training coverage on PubMed (30×200). (a) The $(n\log n)$ regime (orange) achieves near-dense accuracy at 33% coverage across all sampling strategies. (b) Zooming into $(n\log n)$: the sweet spot occurs at $C \in [2,3]$, balancing coverage and accuracy. Below $C=1$, performance degrades rapidly as connectivity becomes harder to maintain. All sparse regimes enforce $d$-core $\geq$ 3.

Theorems & Definitions (7)

  • Lemma 2.1: Binary optimal critic for total variation
  • Proposition 4.1: Quadratic (Gini) projection onto the additive manifold
  • Definition 4.2: Rectangle Deviation
  • Lemma E.1: TVD Self-Information Equals Gini Entropy
  • proof
  • Lemma F.1: Discrete Integrability
  • proof