Exploring the features used for summary evaluation by Human and GPT
Zahra Sadeghi, Evangelos Milios, Frank Rudzicz
TL;DR
The paper investigates which features drive judgments when evaluating textual summaries by humans and GPT-based evaluators, formalizing a correlational framework to link feature-derived scores with scores from both sources. It introduces an unsupervised feature set spanning readability, information-theoretic metrics, and embedding-based similarity, and tests these on the SummEval dataset. A key finding is that conditional perplexity (an uncertainty-driven metric) best aligns with both human and GPT assessments across relevance and coherence, while readability and word-level features differently influence each respondent; prompting GPTs with metric hints further improves alignment. These results shed light on the internal cues used by LLM evaluators and suggest practical prompt strategies to enhance automatic summary quality evaluation.
Abstract
Summary assessment involves evaluating how well a generated summary reflects the key ideas and meaning of the source text, requiring a deep understanding of the content. Large Language Models (LLMs) have been used to automate this process, acting as judges to evaluate summaries with respect to the original text. While previous research investigated the alignment between LLMs and Human responses, it is not yet well understood what properties or features are exploited by them when asked to evaluate based on a particular quality dimension, and there has not been much attention towards mapping between evaluation scores and metrics. In this paper, we address this issue and discover features aligned with Human and Generative Pre-trained Transformers (GPTs) responses by studying statistical and machine learning metrics. Furthermore, we show that instructing GPTs to employ metrics used by Human can improve their judgment and conforming them better with human responses.