Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models

TL;DR

This work addresses the gap between conditional-independence label aggregation and the dense dependencies found among LLM-based judges. It develops a hierarchy of dependence-aware models based on Ising graphical models, deriving Bayes posteriors that are quadratic in votes for class-dependent couplings and linear when couplings are shared, and shows that CI-based methods can be strictly suboptimal under dependence. The authors prove separation results—CI predictors can incur nonvanishing excess risk while the Bayes predictor exploiting dependence can achieve vanishing risk as the judge pool grows—and connect these ideas to latent-factor models. Empirically, dependence-aware Ising predictors outperform classical baselines on relevance, toxicity, and summarization tasks across six judge models, with further improvements as training data increases. A practical EM-based inference framework and diagnostic workflow are provided to enable deployment of dependence-aware label aggregation in real-world evaluation pipelines.

Abstract

Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including LLMs used as judges. Most classical methods, e.g., Dawid-Skene or (weighted) majority voting, assume annotators are conditionally independent given the true label $Y\in\{0,1\}$, an assumption often violated by LLM judges due to shared data, architectures, prompts, and failure modes. Ignoring such dependencies can yield miscalibrated posteriors and even confidently incorrect predictions. We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors. For class-dependent Ising models, the Bayes log-odds is generally quadratic in votes; for class-independent couplings, it reduces to a linear weighted vote with correlation-adjusted parameters. We present finite-$K$ examples showing that methods based on conditional independence can flip the Bayes label despite matching per-annotator marginals. We prove separation results demonstrating that these methods remain strictly suboptimal as the number of judges grows, incurring nonvanishing excess risk under latent factors. Finally, we evaluate the proposed method on three real-world datasets, demonstrating improved performance over the classical baselines.

Figures (8)

• Figure 1: Graphical models for LLM-as-a-judge. Conditional independence (CI); (left): judges are independent given $Y$ (represented by lack arrows connecting the Judge LLMs). Class-dependent Ising; (right): judges exhibit pairwise dependence whose pattern can change with the label ($W^{(0)}\neq W^{(1)}$), enabling class information to affect directly the correlations among judges.
• Figure 2: Model hierarchy via set inclusion:$\text{Conditional Independence (CI)} \subset \text{Class-independent\ Ising} \subset \text{Class-dependent Ising}$.
• Figure 3: Effect of varying the number of training samples (left) and number of judges (right) on the test accuracy. Top, middle and bottom rows correspond respectively to Relevance, Toxicity and Summarization datasets. The standard errors are of small width, although they are plotted.
• Figure 4: Ising predictors (Class-Dep. Ising) versus CI-predictors (CI-WMV): The plots on the left and right represent the empirical risk and empirical separation respectively. Top row corresponds to $\beta_1 =2, c =1.5$. Bottom row corresponds to $\beta_1 =5, c =1$. For all plots, $\pi=0.7, \beta_0 = 0.5, h_0 =-0.5$ and $n=1000$. The standard errors are of small width, although they are plotted.
• Figure 5: Latent-factor (Factor) predictor versus CI-predictors (CI-WMV). The plots on the left and right represent the empirical risk and empirical separation respectively. Top row corresponds to $|\lambda|=0.1, \sigma_Z^2 =1$. Bottom row corresponds to $|\lambda|=0.15, \sigma_Z^2 =1.5$. For all plots, $\pi=0.7, a= 0.5, b =1$ and $n=1000$. The standard errors are of small width, although they are plotted.

Theorems & Definitions (19)

• Theorem 1: Nonvanishing Bayes vs. CI Separation for Class-conditional Ising Models
• Theorem 2: Curie-Weiss separation with informative marginals
• Remark 1: Continuity viewpoint
• Proposition 1: Latent-factor $\Rightarrow$ low-rank class-independent Ising couplings to second order
• proof
• Remark 2: Ising on $\pm1$ spins and the choice of label set for $Y$
• Remark 3
• Theorem 3: Asymptotic Bayes-CI separation under an exchangeable logistic-normal factor
• proof
• Theorem 3: Nonvanishing Bayes vs. CI Separation for Class-conditional Ising Models