Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans?

TL;DR

The paper addresses how automatic summarization evaluation metrics and LLM-based judges align with human judgments in non-English languages, focusing on Basque and Spanish. It introduces BASSE, a large-scale dataset with 2,040 human-annotated summaries and five evaluation criteria (coherence, consistency, fluency, relevance, and 5W1H), plus a broader collection of 22,525 article-subhead pairs. By comparing traditional automatic metrics and LLM judges (proprietary vs open-source), the authors show that proprietary LLMs correlate most with human judgments, particularly for coherence and 5W1H, while open-source judges underperform and automatic metrics vary by language and criterion. The work highlights language-specific challenges in evaluation, demonstrates the value of the 5W1H criterion, and calls for truly multilingual judge models to enable reliable non-English summarization evaluation across domains.

Abstract

Studies on evaluation metrics and LLM-as-a-Judge models for automatic text summarization have largely been focused on English, limiting our understanding of their effectiveness in other languages. Through our new dataset BASSE (BAsque and Spanish Summarization Evaluation), we address this situation by collecting human judgments on 2,040 abstractive summaries in Basque and Spanish, generated either manually or by five LLMs with four different prompts. For each summary, annotators evaluated five criteria on a 5-point Likert scale: coherence, consistency, fluency, relevance, and 5W1H. We use these data to reevaluate traditional automatic metrics used for evaluating summaries, as well as several LLM-as-a-Judge models that show strong performance on this task in English. Our results show that currently proprietary judge LLMs have the highest correlation with human judgments, followed by criteria-specific automatic metrics, while open-sourced judge LLMs perform poorly. We release BASSE and our code publicly, along with the first large-scale Basque summarization dataset containing 22,525 news articles with their subheads.

Figures (4)

• Figure 7: Prompt from Prometheus-Eval used for LLM-as-a-Judge evaluation. Note that instruction, response, reference answer, and rubric are placeholders that are filled with specific information for each example.
• Figure 8: Spearman's model-level rank correlation between automatic metrics and human annotations, in Basque (eu) and Spanish (es).
• Figure B.1: Examples of low- and high-quality summaries of a news article in Spanish. Scores are averages of three annotators. Source of the article: El País.
• Figure C.1: Quadratic Cohen's $\kappa$ in Basque R#3