Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration

Yiyue Qian, Shinan Zhang, Yun Zhou, Haibo Ding, Diego Socolinsky, Yi Zhang

TL;DR

This work proposes CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment that consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle.

Abstract

Large Language Models (LLMs) have revolutionized AI-generated content evaluation, with the LLM-as-a-Judge paradigm becoming increasingly popular. However, current single-LLM evaluation approaches face significant challenges, including inconsistent judgments and inherent biases from pre-training data. To address these limitations, we propose CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment. Unlike existing approaches that rely on competitive debate or single-model evaluation, CollabEval emphasizes collaboration among multiple agents with strategic consensus checking for efficiency. Our extensive experiments demonstrate that CollabEval consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle. The framework provides comprehensive support for various evaluation criteria while ensuring efficiency through its collaborative design.

CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration

TL;DR

This work proposes CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment that consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle.

Abstract

Large Language Models (LLMs) have revolutionized AI-generated content evaluation, with the LLM-as-a-Judge paradigm becoming increasingly popular. However, current single-LLM evaluation approaches face significant challenges, including inconsistent judgments and inherent biases from pre-training data. To address these limitations, we propose CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment. Unlike existing approaches that rely on competitive debate or single-model evaluation, CollabEval emphasizes collaboration among multiple agents with strategic consensus checking for efficiency. Our extensive experiments demonstrate that CollabEval consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle. The framework provides comprehensive support for various evaluation criteria while ensuring efficiency through its collaborative design.
Paper Structure (17 sections, 5 figures, 2 tables)

This paper contains 17 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Framework
  4. Initial Evaluation
  5. Multi-Round Discussion
  6. Final Judge Evaluation
  7. Experiments
  8. Experiment Setup
  9. Evaluation Mode.
  10. Datasets.
  11. Baseline.
  12. Experimental Setting.
  13. Performance Discussion
  14. Discussion about criteria-based evaluation.
  15. Discussion about pair-wise comparison evaluation.
  16. ...and 2 more sections

Figures (5)

  • Figure 1: The framework of CollabEval consists of three main phases: (a) Phase 1: Initial Evaluation - Three evaluators independently assess content, providing evaluation results, confidence scores, and justifications. A consensus check is performed; if consensus is reached, final results are returned, otherwise proceeding to Phase 2. (b) Phase 2: Multi-Round Discussion - Evaluators engage in collaborative discussion sharing agreements, disagreements, and justifications. After each round, a consensus check is performed. If consensus is reached, results are returned; if not, the system checks for maximum rounds or unchanged results before proceeding. (c) Phase 3: Final Judgment - When consensus cannot be reached through discussion, a final judge analyzes all previous evaluation results, confidence scores, agreements/disagreements, and justifications to make the ultimate evaluation decision.
  • Figure 2: Accuracy performance Analysis on relevance evaluation.
  • Figure 3: Gap ratio performance analysis on relevance evaluation.
  • Figure 4: Evaluation trends analysis on relevance (left) and coherence (right) evaluations.
  • Figure 5: Accuracy analysis of CollabEval with collaboration or debate mechanisms on SummEval dataset.