PerSEval: Assessing Personalization in Text Summarizers

TL;DR

This paper challenges the necessity of an accuracy leaderboard, suggesting that relying on accuracy-based aggregated results might lead to misleading conclusions and proposes PerSEval, a novel measure that satisfies the required sufficiency condition.

Abstract

Personalized summarization models cater to individuals' subjective understanding of saliency, as represented by their reading history and current topics of attention. Existing personalized text summarizers are primarily evaluated based on accuracy measures such as BLEU, ROUGE, and METEOR. However, a recent study argued that accuracy measures are inadequate for evaluating the degree of personalization of these models and proposed EGISES, the first metric to evaluate personalized text summaries. It was suggested that accuracy is a separate aspect and should be evaluated standalone. In this paper, we challenge the necessity of an accuracy leaderboard, suggesting that relying on accuracy-based aggregated results might lead to misleading conclusions. To support this, we delve deeper into EGISES, demonstrating both theoretically and empirically that it measures the degree of responsiveness, a necessary but not sufficient condition for degree-of-personalization. We subsequently propose PerSEval, a novel measure that satisfies the required sufficiency condition. Based on the benchmarking of ten SOTA summarization models on the PENS dataset, we empirically establish that -- (i) PerSEval is reliable w.r.t human-judgment correlation (Pearson's r = 0.73; Spearman's $ρ$ = 0.62; Kendall's $τ$ = 0.42), (ii) PerSEval has high rank-stability, (iii) PerSEval as a rank-measure is not entailed by EGISES-based ranking, and (iv) PerSEval can be a standalone rank-measure without the need of any aggregated ranking.

Figures (4)

• Figure 1: EGISES Personalization-Accuracy Paradox. The absurd case of high personalization (thereby high user-experience), yet low accuracy.
• Figure 2: Existence of EGISES Personalization-Accuracy Paradox: High (and same; (a)) DEGRESS, yet low accuracy (red line; (b)).
• Figure 3: PSE-ILM Ablation: Effect of $\beta$ on HJ-Corr; Optimal performance at $\beta=1.7$ across all three standard correlation measures (Pearson $r$, Spearman $\rho$, Kendall $\tau$).
• Figure 4: Sample Questionnaire: Six pairs of summaries for a specific document; five pairs are model-generated summaries (each user evaluates five of the ten models) for a specific document, while one pair is user-generated gold reference).

Theorems & Definitions (18)

• Definition 2.1: Personalized Summarization Oracle
• Definition 2.2: Weak Insensitivity-to-Subjectivity
• Definition 2.3: Strong Insensitivity-to-Subjectivity
• Definition 2.4: Summary-level DEGRESS
• Theorem 1
• proof
• Theorem 2: Limitation 1: Lack of Interpretability of Score
• proof
• Theorem 3: Limitation 2: Lack of Interpretability of Lower Bound
• proof