Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM

TL;DR

This paper addresses the challenge of evaluating long, unstructured summaries by introducing SumAutoEval, a fine-grained, machine-readable evaluation framework. It combines a three-step entity extraction pipeline with multi-prompt scoring across four dimensions—Completeness, Correctness, Alignment, and Readability—to produce objective, robust metrics that correlate more strongly with human judgments than traditional methods. The approach is validated on medical note summarization and benchmarked against ROUGE, BARTScore, and G-Eval, showing improved alignment with expert assessments and resilience to prompt or model variation. The work emphasizes practical impact for high-stakes domains and lays groundwork for integrating fine-grained metrics with holistic evaluation to comprehensively assess summary quality.

Abstract

Due to the exponential growth of information and the need for efficient information consumption the task of summarization has gained paramount importance. Evaluating summarization accurately and objectively presents significant challenges, particularly when dealing with long and unstructured texts rich in content. Existing methods, such as ROUGE (Lin, 2004) and embedding similarities, often yield scores that have low correlation with human judgements and are also not intuitively understandable, making it difficult to gauge the true quality of the summaries. LLMs can mimic human in giving subjective reviews but subjective scores are hard to interpret and justify. They can be easily manipulated by altering the models and the tones of the prompts. In this paper, we introduce a novel evaluation methodology and tooling designed to address these challenges, providing a more comprehensive, accurate and interpretable assessment of summarization outputs. Our method (SumAutoEval) proposes and evaluates metrics at varying granularity levels, giving objective scores on 4 key dimensions such as completeness, correctness, Alignment and readability. We empirically demonstrate, that SumAutoEval enhances the understanding of output quality with better human correlation.