Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation

Marco Salmè, Federico Siciliano, Fabrizio Silvestri, Paolo Soda, Rosa Sicilia, Valerio Guarrasi

TL;DR

This work tackles the dual challenge of interpretability and factual accuracy in radiology report generation by introducing CEMRAG, a framework that unifies interpretable visual concepts with multimodal retrieval-augmented generation. It decomposes visual content into clinically meaningful concepts via SpLiCE, retrieves visually similar cases, and uses a hierarchically structured prompt to guide an LLM in generating grounded reports. Across MIMIC-CXR and IU X-ray, and under zero-shot and supervised fine-tuning regimes, CEMRAG consistently outperforms conventional RAG and concept-only baselines on both NLP and clinical accuracy metrics, challenging the notion of a trade-off between transparency and performance. The approach demonstrates that concept-level interpretability can enhance factual grounding while providing explicit visual rationales, offering a practical, modular path toward clinically trustworthy AI-assisted radiology. These findings have potential to improve workflow efficiency and trust in AI-assisted reporting, with broad applicability to other medical imaging domains.

Abstract

Radiology Report Generation (RRG) through Vision-Language Models (VLMs) promises to reduce documentation burden, improve reporting consistency, and accelerate clinical workflows. However, their clinical adoption remains limited by the lack of interpretability and the tendency to hallucinate findings misaligned with imaging evidence. Existing research typically treats interpretability and accuracy as separate objectives, with concept-based explainability techniques focusing primarily on transparency, while Retrieval-Augmented Generation (RAG) methods targeting factual grounding through external retrieval. We present Concept-Enhanced Multimodal RAG (CEMRAG), a unified framework that decomposes visual representations into interpretable clinical concepts and integrates them with multimodal RAG. This approach exploits enriched contextual prompts for RRG, improving both interpretability and factual accuracy. Experiments on MIMIC-CXR and IU X-Ray across multiple VLM architectures, training regimes, and retrieval configurations demonstrate consistent improvements over both conventional RAG and concept-only baselines on clinical accuracy metrics and standard NLP measures. These results challenge the assumed trade-off between interpretability and performance, showing that transparent visual concepts can enhance rather than compromise diagnostic accuracy in medical VLMs. Our modular design decomposes interpretability into visual transparency and structured language model conditioning, providing a principled pathway toward clinically trustworthy AI-assisted radiology.

Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation

TL;DR

This work tackles the dual challenge of interpretability and factual accuracy in radiology report generation by introducing CEMRAG, a framework that unifies interpretable visual concepts with multimodal retrieval-augmented generation. It decomposes visual content into clinically meaningful concepts via SpLiCE, retrieves visually similar cases, and uses a hierarchically structured prompt to guide an LLM in generating grounded reports. Across MIMIC-CXR and IU X-ray, and under zero-shot and supervised fine-tuning regimes, CEMRAG consistently outperforms conventional RAG and concept-only baselines on both NLP and clinical accuracy metrics, challenging the notion of a trade-off between transparency and performance. The approach demonstrates that concept-level interpretability can enhance factual grounding while providing explicit visual rationales, offering a practical, modular path toward clinically trustworthy AI-assisted radiology. These findings have potential to improve workflow efficiency and trust in AI-assisted reporting, with broad applicability to other medical imaging domains.

Abstract

Radiology Report Generation (RRG) through Vision-Language Models (VLMs) promises to reduce documentation burden, improve reporting consistency, and accelerate clinical workflows. However, their clinical adoption remains limited by the lack of interpretability and the tendency to hallucinate findings misaligned with imaging evidence. Existing research typically treats interpretability and accuracy as separate objectives, with concept-based explainability techniques focusing primarily on transparency, while Retrieval-Augmented Generation (RAG) methods targeting factual grounding through external retrieval. We present Concept-Enhanced Multimodal RAG (CEMRAG), a unified framework that decomposes visual representations into interpretable clinical concepts and integrates them with multimodal RAG. This approach exploits enriched contextual prompts for RRG, improving both interpretability and factual accuracy. Experiments on MIMIC-CXR and IU X-Ray across multiple VLM architectures, training regimes, and retrieval configurations demonstrate consistent improvements over both conventional RAG and concept-only baselines on clinical accuracy metrics and standard NLP measures. These results challenge the assumed trade-off between interpretability and performance, showing that transparent visual concepts can enhance rather than compromise diagnostic accuracy in medical VLMs. Our modular design decomposes interpretability into visual transparency and structured language model conditioning, providing a principled pathway toward clinically trustworthy AI-assisted radiology.
Paper Structure (26 sections, 3 equations, 3 figures, 6 tables)

This paper contains 26 sections, 3 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Interpretability for Vision-Language Models
  4. Multimodal Retrieval-Augmented Generation in Medicine
  5. Limitations and Motivations
  6. Methodology
  7. Visual Encoding and Projection
  8. Concept Extraction
  9. Multimodal Retrieval Augmented Generation
  10. Hierarchical Prompt Construction and Report Generation
  11. Experimental Setup
  12. Datasets
  13. Model Configurations and Experimental Conditions
  14. Evaluation Metrics
  15. Results
  16. ...and 11 more sections

Figures (3)

  • Figure 1: CEMRAG overall framework, combining interpretable concept extraction with RAG for transparent and accurate radiology reporting.
  • Figure 2: SpLiCE performance across vocabulary sizes and $\lambda$ values. The figure reports three complementary metrics: Left: precision of extracted concepts; Center: cosine similarity between the original CLIP embedding and its sparse reconstruction; Right: average number of active concepts (sparsity). The results highlight the trade-off between fidelity, interpretability, and terminological precision.
  • Figure 3: Precision as a function of the number of selected concepts $\tau \in \{3, 5, 7\}$ for both the 100- and 200-bigram vocabularies. Precision decreases monotonically with increasing $\tau$, reflecting the progressive inclusion of lower-ranked concepts with weaker activation strengths. The 100-bigram vocabulary maintains a consistent advantage across all $\tau$ values, indicating that the highest-confidence concepts are highly informative for guiding retrieval-augmented report generation.