Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge

TL;DR

CCE introduces crowd-based comparative evaluation to overcome CoT limitations in LLM-based judgments by generating diverse crowd responses, selecting informative crowd judgments through Criticizing Selection and Outcome Removal, and using crowd-informed context to produce richer CoT judgments. The framework supports distillation of high-quality CoTs for training smaller judges and extends to crowd rejection sampling in SFT, yielding improvements across multiple preference benchmarks and downstream tasks. Empirical results show a $6.7\%$ average accuracy gain across five benchmarks, with substantial gains in reward evaluation, bias robustness, and SFT data efficiency, alongside evidence that CoTs become more comprehensive and better aligned with human judgments as inference scales. Overall, CCE offers a scalable, reliable auto-evaluation paradigm that enhances CoT reasoning, judge distillation, and data selection for LLM alignment.

Abstract

LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become a widely adopted auto-evaluation method. However, its reliability is compromised by the CoT reasoning's inability to capture comprehensive and deeper details, often leading to incomplete outcomes. Existing methods mainly rely on majority voting or criteria expansion, which is insufficient to address the limitation in CoT. We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses, thereby exposing deeper and more comprehensive details within the candidate responses. This process effectively guides LLM-as-a-Judge to provide a more detailed CoT judgment. Extensive experiments demonstrate that our approach enhances evaluation reliability, achieving an average accuracy gain of 6.7% across five benchmarks. Moreover, our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling for supervised fine-tuning (SFT), referred to as crowd rejection sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs generated by ours are more comprehensive and of higher quality, and evaluation accuracy improves as inference scales.

Figures (8)

• Figure 1: An overview of our method. By evaluating the candidate responses A/B alongside the crowd responses, the resulting crowd judgment can be used as context to enrich the evaluation of A/B responses, leading to a more comprehensive CoT judgment.
• Figure 2: Pipeline of our proposed crowd-based comparative evaluation. For a given instance $(x, y^A, y^B)$, we first use the LLM to generate crowd responses $\left\{y^i|i\in\{C,D,E,...\}\right\}$ based on $x$. These responses are then compared with $y^A$ and $y^B$ to produce initial crowd judgments $\mathcal{J}$, which are subsequently refined into $\hat{\mathcal{J}}$ after selection and processing. Finally, $\hat{\mathcal{J}}$ are used as contextual input to evaluate the instance $(x, y^A, y^B)$.
• Figure 3: Evaluation performance under scaling crowd judgments in the context. As the number of crowd judgments grows, both accuracy and CoT length generally increase.
• Figure 4: CoT Comparison. CCE’s CoT consistently yields a higher average number of key points and a higher coverage rate across all benchmarks.
• Figure 5: Prompt of Our Method.