What's under the hood: Investigating Automatic Metrics on Meeting Summarization

TL;DR

This work targets the misalignment between automatic evaluation metrics and human judgments in English meeting summarization. By annotating QMSum transcripts for domain-specific challenges and errors and evaluating nine metrics across encoder-decoder and decoder-only models, the authors map how various metrics reflect or mask different errors. The study shows that no single metric captures all error types, with many metrics failing to reflect error severity, and highlights the potential of LLM-based evaluation as a promising direction. Overall, the findings argue for more nuanced, task-aware evaluation frameworks and share a reusable benchmarking resource to spur advances in meeting-summarization evaluation.

Abstract

Meeting summarization has become a critical task considering the increase in online interactions. While new techniques are introduced regularly, their evaluation uses metrics not designed to capture meeting-specific errors, undermining effective evaluation. This paper investigates what the frequently used automatic metrics capture and which errors they mask by correlating automatic metric scores with human evaluations across a broad error taxonomy. We commence with a comprehensive literature review on English meeting summarization to define key challenges like speaker dynamics and contextual turn-taking and error types such as missing information and linguistic inaccuracy, concepts previously loosely defined in the field. We examine the relationship between characteristic challenges and errors by using annotated transcripts and summaries from Transformer-based sequence-to-sequence and autoregressive models from the general summary QMSum dataset. Through experimental validation, we find that different model architectures respond variably to challenges in meeting transcripts, resulting in different pronounced links between challenges and errors. Current default-used metrics struggle to capture observable errors, showing weak to mid-correlations, while a third of the correlations show trends of error masking. Only a subset reacts accurately to specific errors, while most correlations show either unresponsiveness or failure to reflect the error's impact on summary quality.