PCoA: A New Benchmark for Medical Aspect-Based Summarization With Phrase-Level Context Attribution

TL;DR

PCoA addresses the need for verifiable medical aspect-based summaries by tying outputs to precise source context. It introduces 1,799 annotated summaries across 152 randomized controlled melanoma trials, with phrase-level context attribution and a decoupled evaluation protocol. The paper provides a robust benchmark, an evaluation algorithm, and a comparative LLM study showing prior-context attribution improves citation and phrase grounding. The dataset and code are publicly available to support reproducibility and future research.

Abstract

Verifying system-generated summaries remains challenging, as effective verification requires precise attribution to the source context, which is especially crucial in high-stakes medical domains. To address this challenge, we introduce PCoA, an expert-annotated benchmark for medical aspect-based summarization with phrase-level context attribution. PCoA aligns each aspect-based summary with its supporting contextual sentences and contributory phrases within them. We further propose a fine-grained, decoupled evaluation framework that independently assesses the quality of generated summaries, citations, and contributory phrases. Through extensive experiments, we validate the quality and consistency of the PCoA dataset and benchmark several large language models on the proposed task. Experimental results demonstrate that PCoA provides a reliable benchmark for evaluating system-generated summaries with phrase-level context attribution. Furthermore, comparative experiments show that explicitly identifying relevant sentences and contributory phrases before summarization can improve overall quality. The data and code are available at https://github.com/chubohao/PCoA.

Figures (15)

• Figure 1: Illustrative example from the PCoA benchmark. Given an article and a target aspect, the task is to generate an aspect-based summary with cited contextual sentences and aligned contributory phrases.
• Figure 2: Human evaluation results assessing the completeness (top) and conciseness (bottom) of summaries, cited sentences, and contributory phrases across sixteen aspects, as measured using a 5-point Likert scale.
• Figure 3: Overview of the automatic evaluation framework. Summaries are evaluated using Claim Recall and Claim Precision (left), citations are assessed with Citation Recall and Citation Precision (middle), and contributory phrases are evaluated using Phrase Recall and Phrase Precision (right).
• Figure 4: Effects of the number of reference subclaims, citations, and contributory phrases on model performance. The x-axis represents the count, while the y-axis indicates the metric scores.
• Figure 5: