SkillAggregation: Reference-free LLM-Dependent Aggregation

TL;DR

SkillAggregation tackles the challenge of combining multiple LLM judgments without ground-truth labels by introducing a context-aware, reference-free aggregation framework that extends Crowdlayer with trainable skill-estimate vectors and a bottleneck context encoder. It trains to predict LLM judgments and performs Bayesian-style posterior inference at test time, achieving superior accuracy over baselines on HaluEval-Dialogue, TruthfulQA, and Chatbot Arena, with notable gains on 7B/8B and 70B models. The approach demonstrates robustness to context encoders, reveals interpretable relationships between learned skills and judge accuracy, and remains effective under dataset size variations and common biases, highlighting its practical impact for scalable, reliable LLM evaluation.

Abstract

Large Language Models (LLMs) are increasingly used to assess NLP tasks due to their ability to generate human-like judgments. Single LLMs were used initially, however, recent work suggests using multiple LLMs as judges yields improved performance. An important step in exploiting multiple judgements is the combination stage, aggregation. Existing methods in NLP either assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection. This work focuses on aggregating predictions from multiple systems where no reference labels are available. A new method called SkillAggregation is proposed, which learns to combine estimates from LLM judges without needing additional data or ground truth. It extends the Crowdlayer aggregation method, developed for image classification, to exploit the judge estimates during inference. The approach is compared to a range of standard aggregation methods on HaluEval-Dialogue, TruthfulQA and Chatbot Arena tasks. SkillAggregation outperforms Crowdlayer on all tasks, and yields the best performance over all approaches on the majority of tasks.

Figures (10)

• Figure 1: Illustration of the worker aggregation process. LLM judges observe context $X_n$ and provide judgments ${\mathbf{y}}_n \in [0,1]^K$ for ground truth ${\textnormal{c}}_n \in \{0,1\}$. An aggregation method takes the following as input: a dataset $\{ (X_n, {\mathbf{y}}_n) \}_{n=1}^N$, and produces group estimates $\{\hat{c}_n \in \{0,1\}\}_{n=1}^N$ of the judgments.
• Figure 2: The SkillAggregation model with an example context from the TruthfulQA dataset. The terms $p_{0}^{(n,k)}, p_{1}^{(n,k)}$ are estimated skills for $k$-th LLM, and $s_{n,0}, s_{n,1}$ are outputs from the bottleneck layer. The outputs $\hat{y}_{n,k}$ are the predicted LLM judgments for the $k$-th LLM on data sample $n$.
• Figure 3: Scatter plot of the slope $(\hat{p}_{0}^{(n,k)} + \hat{p}_{1}^{(n,k)} - 1)$ representing the skill of each LLM from SkillAggregation against accuracies of each LLM on HaluEval. PCC stands for Pearson Correlation Coefficient.
• Figure 4: Accuracies of SkillAggregation and DawidSkene against majority voting on different subsets of LLM judges on TruthfulQA. The equal-performance line is plotted, and points on the upper-left side indicate improved performance compared to majority voting.
• Figure 5: Accuracy when replacing the GPT-2 context encoders with RoBERTa and Gemma-2B for Crowdlayer and SkillAggregation on TruthfulQA and Chatbot Arena datasets.