Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator

TL;DR

Evaluating meeting summaries with traditional metrics often fails to match human judgments and incurs high labeling costs. The authors introduce MESA, a multi-stage, LLM-based evaluator that combines per-error three-step assessments, a multi-agent debate protocol (MADP), and a feedback-driven self-training loop to align scores with human judgments and accommodate task-specific error definitions. Using GPT-4o, MESA achieves mid-to-high correlations with human judgment for error existence and better reflection of error impact than prior evaluators, with improvements of roughly 0.25 in correlation on average. The framework is designed to be adaptable to custom error guidelines and requires limited labeled data, with plans to release code and datasets to foster broader research and practical deployment in meeting-summarization evaluation.

Abstract

The quality of meeting summaries generated by natural language generation (NLG) systems is hard to measure automatically. Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. Recent studies suggest using large language models (LLMs), which have the benefit of better context understanding and adaption of error definitions without training on a large number of human preference judgments. However, current LLM-based evaluators risk masking errors and can only serve as a weak proxy, leaving human evaluation the gold standard despite being costly and hard to compare across studies. In this work, we present MESA, an LLM-based framework employing a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. We show that MESA's components enable thorough error detection, consistent rating, and adaptability to custom error guidelines. Using GPT-4o as its backbone, MESA achieves mid to high Point-Biserial correlation with human judgment in error detection and mid Spearman and Kendall correlation in reflecting error impact on summary quality, on average 0.25 higher than previous methods. The framework's flexibility in adapting to custom error guidelines makes it suitable for various tasks with limited human-labeled data.

Figures (5)

• Figure 1: Architecture of MESA displaying the singel-aspect assessment using three stages and the self-training mechanic for feedback-based alignment improvement with available human data.
• Figure 2: Multi-agent discussion protocol used, consisting of an initial draft generator, three synchronously acting challengers, and a moderator summarizing the individual statements into a final task solution.
• Figure 3: The prompt template used to task an LLM instance to identify potential error instances.
• Figure 4: The prompt template used to task an LLM instance to rate detected error instance.
• Figure 5: The prompt template used to task an LLM instance to perform the scoring.