Cite-While-You-Generate: Training-Free Evidence Attribution for Multimodal Clinical Summarization

TL;DR

Trustworthy clinical summarization requires transparent provenance for generated claims. The paper introduces Cite-While-You-Generate, a training-free attention-guided attribution framework that provides sentence- and image-level citations during generation by leveraging decoder attentions. It supports two multimodal grounding modes—RAW image attention and caption-as-image-span—evaluated on CliConSummation and MIMIC-CXR with open models, outperforming embedding-based and self-attribution baselines and revealing trade-offs between grounding fidelity and deployment practicality. The findings show that attention-based attribution improves interpretability and trust, offering a viable path toward deployable, provenance-rich clinical summarization systems in settings where images may be shielded or unavailable.

Abstract

Trustworthy clinical summarization requires not only fluent generation but also transparency about where each statement comes from. We propose a training-free framework for generation-time source attribution that leverages decoder attentions to directly cite supporting text spans or images, overcoming the limitations of post-hoc or retraining-based methods. We introduce two strategies for multimodal attribution: a raw image mode, which directly uses image patch attentions, and a caption-as-span mode, which substitutes images with generated captions to enable purely text-based alignment. Evaluations on two representative domains: clinician-patient dialogues (CliConSummation) and radiology reports (MIMIC-CXR), show that our approach consistently outperforms embedding-based and self-attribution baselines, improving both text-level and multimodal attribution accuracy (e.g., +15% F1 over embedding baselines). Caption-based attribution achieves competitive performance with raw-image attention while being more lightweight and practical. These findings highlight attention-guided attribution as a promising step toward interpretable and deployable clinical summarization systems.

Figures (2)

• Figure 1: Overview of our proposed framework for inference-time source attribution in clinical summarization. Given a dialogue and an associated image, our method produces summaries with explicit citations to supporting evidence. We support two modes within the same framework: in the IMG_RAW mode, decoder attentions over image patches are directly aggregated to attribute visual evidence, whereas in the IMG_CAP mode, the image is converted into a one-sentence caption to enable purely text-based alignment. Each generated summary sentence is annotated with citations to source text spans and/or the image, providing interpretable and trustworthy clinical summaries.
• Figure 2: Ablation on text-only attribution.(A) Majority vs. Max (best $\tau$ per $k$): majority voting yields 60-point higher Macro-F1 and far stronger exact match than max, which collapses to near-random performance. (B) Effect of $k$ under majority voting: attribution quality peaks around $\tau=0.2$, with $k=3$ providing the best balance between robustness and stability. (C) Fine-grained $\tau$ sweep for $k=3$, majority shows the optimal range lies between $0.15$ and $0.18$, supporting our choice of $\tau=0.16$ as the default setting. Full numeric results are provided in Appendix \ref{['appendix:ablation']}.