ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation

TL;DR

The paper tackles the hard problem of evaluating personalized long-form text generation by proposing ExPerT, an explainable reference-based evaluation framework. ExPerT decomposes outputs into atomic aspects with evidences, uses an LLM to match aspects and assess content and writing-style alignment, and provides per-decision rationales to enhance transparency. Empirical results on the LongLaMP benchmark show ExPerT achieves the best alignment with human judgments (0.74) and demonstrates robustness to manipulation and favorable efficiency, with a human-judged explanation quality of 4.7/5. The work offers a reproducible, interpretable evaluation approach tailored for personalization and releases code to support future research and application.

Abstract

Evaluating personalized text generated by large language models (LLMs) is challenging, as only the LLM user, i.e., prompt author, can reliably assess the output, but re-engaging the same individuals across studies is infeasible. This paper addresses the challenge of evaluating personalized text generation by introducing ExPerT, an explainable reference-based evaluation framework. ExPerT leverages an LLM to extract atomic aspects and their evidence from the generated and reference texts, match the aspects, and evaluate their alignment based on content and writing style -- two key attributes in personalized text generation. Additionally, ExPerT generates detailed, fine-grained explanations for every step of the evaluation process, enhancing transparency and interpretability. Our experiments demonstrate that ExPerT achieves a 7.2% relative improvement in alignment with human judgments compared to the state-of-the-art text generation evaluation methods. Furthermore, human evaluators rated the usability of ExPerT's explanations at 4.7 out of 5, highlighting its effectiveness in making evaluation decisions more interpretable.

Figures (9)

• Figure 1: ExPerT pipeline: Generated and reference outputs are first decomposed into atomic aspects along with their corresponding evidences. Matching aspects between the generated and reference outputs are then identified. Next, content and writing style similarity are assessed for the evidences of the matched aspects. Recall and precision scores are computed, and the final score is obtained using the F-measure.
• Figure 2: The prompts used for aspect extraction, aspect, content, and writing style matching in ExPerT.
• Figure 3: The alignment between ExPerT different methods for content and style score aggregation with human judgment in evaluation.
• Figure 4: The alignment between ExPerT with different LLMs and sizes with human judgment in evaluation.
• Figure 5: The average ExPerT score across varying percentages of examples in the dataset randomly substituted with random profiles from other users.