Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process

TL;DR

This work tackles the problem of leveraging multiple open LLMs without labeled data by introducing LLM-PeerReview, an unsupervised, peer-review–inspired ensemble framework. It organizes the ensemble into three stages: scoring candidate responses via LLMs as judges (with a bias-mitigating flipped-triple trick), reasoning over scores through either simple averaging or a Dawid–Skene–style truth inference, and selecting the top-scoring response. The authors formalize a graphical model with latent truth scores and optimize it with EM, yielding a principled, reliability-aware aggregation of judgments. Empirical results on TriviaQA, GSM8k, MATH, and AlpacaEval show that LLM-PeerReview variants consistently surpass single LLMs and strong baselines, with notable improvements over Smoothie-Global, highlighting the practical impact of integrating diverse model judgments in an interpretable, unsupervised framework.

Abstract

We propose LLM-PeerReview, an unsupervised LLM Ensemble method that selects the most ideal response from multiple LLM-generated candidates for each query, harnessing the collective wisdom of multiple models with diverse strengths. LLM-PeerReview is built on a novel, peer-review-inspired framework that offers a clear and interpretable mechanism, while remaining fully unsupervised for flexible adaptability and generalization. Specifically, it operates in three stages: For scoring, we use the emerging LLM-as-a-Judge technique to evaluate each response by reusing multiple LLMs at hand; For reasoning, we can apply a principled graphical model-based truth inference algorithm or a straightforward averaging strategy to aggregate multiple scores to produce a final score for each response; Finally, the highest-scoring response is selected as the best ensemble output. LLM-PeerReview is conceptually simple and empirically powerful. The two variants of the proposed approach obtain strong results across four datasets, including outperforming the recent advanced model Smoothie-Global by 6.9% and 7.3% points, respectively.

Figures (16)

• Figure 1: The proposed LLM-PeerReview contains three steps: (1) Scoring: For a given query, after each LLM independently generates a response (analogous to a submitted academic paper), LLM-PeerReview applies the LLM-as-a-Judge technique (and the proposed flipped-triple scoring trick), treating each model as a reviewer to assign scores to all candidate responses; (2) Reasoning: LLM-PeerReview then uses a truth inference algorithm—analogous to a senior reviewer—to estimate a final score for each response. (Notably, for the variant LLM-PeerReview-Weighted, the inference algorithm is performed using score information across all queries, allowing the model to learn each LLM’s scoring behavior using global information from the dataset, thereby enabling fine-grained, reliability-aware score aggregation); (3) Selecting the best: Finally, for each query, LLM-PeerReview selects the response with the highest final score as the ensemble output—analogous to how a senior reviewer chooses the best paper from a specific submission pool.
• Figure 2: Probabilistic graphical representation.
• Figure 3: LLM performances (bottom: AlpacaEval).
• Figure 4: Left: The transition matrix of each LLM estimated by LLM-PeerReview-Weighted. Right: Correlation between matrix diagonal information of each LLM and its performance as a single judge (corresponding to "our variants" in Table \ref{['table: main results']}). For the first three datasets with ground-truth answers, diagonal information is represented by extreme values $(\pi_{1 1}^{(j^{\prime})} + \pi_{K K}^{(j^{\prime})})$; for the instruction-following dataset AlpacaEval, the sum of all diagonal values is used.
• Figure 5: Performance of various variants across different scoring levels.