Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges

TL;DR

This work tackles the challenge of evaluating multi-turn dialogues by biases in LLM judges and high inference costs from multi-judge ensembles. It introduces MTDEval, an encoder-based, lightweight evaluator that learns from multiple LLM judges using a probabilistic, maximum-likelihood framework grounded in Thurstone's Case V, jointly modeling dialogue quality and judge reliability. The authors build P2-MTD, a large-scale multi-judge preference dataset, and Daily-MTD, a human-annotated benchmark, then show MTDEval achieves superior robustness and efficiency across single-rating, pairwise, and multi-dimensional evaluation tasks, outperforming open-source baselines and approaching or exceeding several proprietary models in many settings. The approach enables fast, fine-grained dialogue quality assessment with reduced computational cost, and the released resources support broader research in reliable automatic evaluation of multi-turn dialogues.

Abstract

Evaluating the conversational abilities of large language models (LLMs) remains a challenging task. Current mainstream approaches primarily rely on the "LLM-as-a-judge" paradigm, where an LLM is prompted to serve as an evaluator to assess dialogue quality. However, such methods often suffer from various biases, which undermine the reliability and consistency of the evaluation results. To mitigate these biases, recent methods employ multiple LLMs as judges and aggregate their judgments to select the optimal assessment. Although effective, this multi-judge approach incurs significant computational overhead during inference. In this paper, we propose an efficient dialogue evaluator that captures the collective wisdom of multiple LLM judges by aggregating their preference knowledge into a single model. Our approach preserves the advantages of diverse multi-judge feedback while drastically reducing the evaluation cost, enabling fast, flexible, and fine-grained dialogue quality assessment. Extensive experiments on seven single rating and pairwise comparison dialogue evaluation benchmarks demonstrate that our method outperforms existing baselines across diverse scenarios, showcasing its efficiency and robustness.

Figures (4)

• Figure 1: Comparison among different dialogue evaluation paradigms. (a) The traditional LLM-as-a-Judge approach. (b) Learning an evaluator using the preference data from a judge. (c) The aggregation of multiple LLMs for evaluation. (d) Learning an evaluator from multiple LLM judges (Ours).
• Figure 2: Overview of MTDEval. The left part illustrates the construction of a multi-judge-annotated, fine-grained preference dataset used for training. The right part demonstrates the model architecture and training procedure, which comprises an LLM-based text embedding model and an MLP-based quality prediction head. The training involves a probabilistic formulation of pairwise preferences with judge reliability prediction, which is optimized by maximum likelihood estimation.
• Figure 3: Performance comparison of models trained on annotations from individual LLM judges versus our model trained on multi-judge preferences.
• Figure 4: The learned sensitivity ($\alpha$) and specificity ($\beta$) of the five LLM judges.