Becoming Experienced Judges: Selective Test-Time Learning for Evaluators

TL;DR

The paper tackles the rigidity of LLM-based evaluators that judge samples independently by introducing Learning While Evaluating (LWE), which maintains a meta-prompt that evolves across test cases to produce sample-specific evaluation criteria and incorporate self-feedback. A selective variant, SelectiveLWE, updates the meta-prompt only on inconsistent (challenging) cases to reduce computational cost. Empirical results on two multimodal pairwise benchmarks show that SelectiveLWE improves accuracy and consistency while constraining inference cost, and that the approach generalizes across different evaluators. The work demonstrates that evaluators can effectively learn from experience during deployment, enabling more reliable, sample-tailored judgments without additional training data. Limitations include reliance on base model capabilities and remaining challenges for truly open/open-ended settings, suggesting avenues for extending the framework to broader evaluation tasks.

Abstract

Automatic evaluation with large language models, commonly known as LLM-as-a-judge, is now standard across reasoning and alignment tasks. Despite evaluating many samples in deployment, these evaluators typically (i) treat each case independently, missing the opportunity to accumulate experience, and (ii) rely on a single fixed prompt for all cases, neglecting the need for sample-specific evaluation criteria. We introduce Learning While Evaluating (LWE), a framework that allows evaluators to improve sequentially at inference time without requiring training or validation sets. LWE maintains an evolving meta-prompt that (i) produces sample-specific evaluation instructions and (ii) refines itself through self-generated feedback. Furthermore, we propose Selective LWE, which updates the meta-prompt only on self-inconsistent cases, focusing computation where it matters most. This selective approach retains the benefits of sequential learning while being far more cost-effective. Across two pairwise comparison benchmarks, Selective LWE outperforms strong baselines, empirically demonstrating that evaluators can improve during sequential testing with a simple selective update, learning most from the cases they struggle with.

Figures (24)

• Figure 1: Comparison of three evaluation approaches. A vanilla judge evaluates each test case (TC) independently using a fixed prompt throughout evaluation (eval.). LWE (ours) employs a meta-prompt (M) that evolves sequentially as the evaluator progresses through test cases, enabling sample-specific tailoring and continual improvement during evaluation. SelectiveLWE (ours) further enhances efficiency by updating the meta-prompt only on challenging cases (e.g., the red-highlighted TCs 2, 5, 8, 14), preserving performance gains while substantially reducing computational overhead. The color gradient illustrates progressive improvement of the judge’s performance over time.
• Figure 2: Overview of Learning While Evaluating (LWE). Given a test case $x_t$, the meta-prompt $M_{t-1}$ generates a sample-specific evaluation prompt $P_t$, which the evaluator uses to produce a judgment $y_t$. The evaluator then reflects on its decision to produce self-feedback $f_t$, which is incorporated into the meta-prompt to form $M_t$ for subsequent cases.
• Figure 3: Cumulative accuracy over test cases. Curves are computed on the vanilla-inconsistent subsets of each benchmark, where SelectiveLWE performs updates. SelectiveLWE maintains consistently higher accuracy than both the vanilla baseline and the Sample-Specific Prompt baseline as evaluation progresses, illustrating the benefits of sequential learning and the model’s capacity to integrate insights acquired during evaluation. Gray-shaded areas indicate confidence intervals for the vanilla baseline, computed at each point using the binomial proportion method with significance level $\alpha = 0.05$.
• Figure 4: Effect of inconsistency ratio on meta-prompt updates. We evaluate LWE on subsets containing different proportions of inconsistent samples, where inconsistency is computed based on vanilla predictions. As the inconsistency ratio increases, LWE shows larger performance gains over the vanilla baseline (purple vs. gray), highlighting its effectiveness in handling inconsistent cases. For each benchmark, the total number of samples is fixed: 248 for VLRewardBench and 137 for MMRewardBench.
• Figure 5: Illustration of how the meta-prompt evolves after a single refinement step. Red and blue text denotes instructions removed and added during the update, reflecting the shift from loosely specified checks to clearer, more structured heuristics under LWE. The full meta-prompts are provided in Appendix \ref{['appendix:fig7_full_prompts']}.