Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling

TL;DR

This work provides an analytically tractable framework to study inference-time scaling for LLMs by casting LLM-as-a-judge as a reward-weighted Bayesian regression problem. In the high-dimensional limit, deterministic equivalents yield closed-form predictive statistics, enabling precise analysis of how the number of inference-time samples $k$ and the reward temperature $T$ affect generalization, including when more inference-time compute helps or hurts. Key results show monotonic improvement with $k$ when the reward aligns with the teacher, but potential finite-$k$ optima and an optimal temperature emerge under misalignment; in the best-of-$k$ regime, generalization decays as $\Theta(1/k^2)$ with a coefficient that can be computed via extreme-value theory. The theory is supported by LLM experiments that reveal analogous behaviors, and the work also clarifies trade-offs between training and inference-time compute under varying task difficulty and reward quality, offering actionable guidance for scaling inference-time computation.

Abstract

Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time. However, the principles behind inference time scaling are not well understood. In this paper, we introduce an analytically tractable model of inference-time scaling: Bayesian linear regression with a reward-weighted sampler, where the reward is determined from a linear model, modeling LLM-as-a-judge scenario. We study this problem in the high-dimensional regime, where the deterministic equivalents dictate a closed-form expression for the posterior predictive mean and variance. We analyze the generalization error when training data are sampled from a teacher model. We draw $k$ inference-time samples and select via softmax at a temperature applied to a quadratic reward. When the reward is not too different from the teacher, the generalization error decreases monotonically with increasing inference time samples $k$. However, the specific reward that optimizes inference-time selection generally differs from the teacher. In contrast, substantial reward misspecification induces a finite optimal $k$ beyond which more sampling can increase the generalization error. For fixed $k$, there exists an optimal sampling temperature. We experimentally verify these facts in large language model inference with an additional large language model as a judge. In the "best-of-$k$" limit with the teacher as reward, we theoretically show that the generalization error decays as $Θ(1/k^2)$ and determine the leading coefficient via extreme value theory. These formulas delineate domains where scaling inference-time computation is provably preferable to collecting more data. Finally, we demonstrate that when task difficulty increases, the previously mentioned advantage of inference-time compute degrades.

Figures (12)

• Figure 1: In the plot the radial distance is the magnitude $c$ of the vector $\mathbf w_R-\mathbf w_T$ and the polar variable is the angle $\theta$ between $\mathbf w_R-\mathbf w_T$ and $\mathbf w_T$. We have chosen $S=1, \sigma=10^{-4},\gamma= 10^{-3}, d=2, n=10^4$ and sampled teacher weight $\mathbf w_T=(c \cos \theta_T, c \sin \theta_T ) \sim \mathcal{N}(0, 2^2 \mathbf I)$. We have parameterized the reward weight as follows: $\mathbf w_R=\mathbf w_T+(c \cos (\theta_T+\theta), c \sin (\theta_T+\theta) ), \theta \in [0, 2\pi)$. See section \ref{['setup']} for details of notation and conventions. From the plot, we see that as temperature $T$ decreases the domain, where generalization error $\delta$ decreases monotonically with the increase in inference-time samples $k$, shrinks. Similar result is presented in remark \ref{['opt_reward']} for the proportional limit.
• Figure 2: In the plot we have chosen $S=1, \sigma=10^{-4},\gamma= 10^{-3}, n=10^4, d=10^1$ and sampled teacher weight $\mathbf w_T \sim \mathcal{N}(0, 2^2 \mathbf I)$. We have parameterized the reward weight as follows: $\mathbf w_R=(1+cR/(R+S^2))\mathbf w_T$. Solid and dashed lines correspond to the experimental results and the formula in Result \ref{['thm:DE']} respectively. On y-axis we plot generalization error normalized by its natural scale set by the noise $\delta/\sigma^2$ and in x-axis we plot the logarithm of the number of inference time samples $\log k$.
• Figure 3: In the plot we have chosen $S=1, \sigma=10^{-4},\gamma=10^{-3},n=10^4, d=10$, $k=50$ and used the following parameterization $\mathbf w_R=(1+cR/(R+S^2))\mathbf w_T$ and sampled $\mathbf w_T \sim \mathcal{N}(0, 2^2 \mathbf I)$. This plot shows dependence of $\delta$ on $c$ for various values of $T$ at fixed $k$. We see that $\delta$ is minimized at $c \approx T/(2\sigma^2)$ as expected from Remark \ref{['opt_reward']}.
• Figure 4: In the plot we have chosen $S=1, \sigma=10^{-4},\gamma=10^{-3},n=10^4, d=10$, $T=200 \sigma^2$ and used the following parameterization $\mathbf w_R=(1+cR/(R+S^2))\mathbf w_T$ and sampled $\mathbf w_T \sim \mathcal{N}(0, 2^2 \mathbf I)$. We plot the scaled value of $\delta-\delta_\infty, \delta_\infty \approx \delta_{k=100}$ as a function of $k$ for various values of $c$. This shows existence of an optimal value of $k$ - theoretical prediction for it is denoted as $k_{opt}$ as given in Remark \ref{['opt_k']}.
• Figure 5: In the plot we have chosen $S=1, \sigma=10^{-4},\gamma=10^{-3},n=10^4, d=10$ and used the following parameterization $\mathbf w_R=(1+cR/(R+S^2))\mathbf w_T$ and sampled $\mathbf w_T \sim \mathcal{N}(0, 2^2 \mathbf I)$. This plot shows existence of an optimal value of temperature $T$ at fixed $k=50$. For the optimal value, we find reasonable agreement with the theoretical prediction in in Remark \ref{['opt_temp']}.

Theorems & Definitions (9)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Remark 5
• Remark 6
• Definition 5: Maximum domain of attraction
• Remark 7: Characterisation via exceedance rates
• Remark 8