Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Multi-Agent Framework for Radiology Report Generation

Ziruo Yi, Ting Xiao, Mark V. Albert

TL;DR

This work tackles radiology report generation (RRG) by introducing a multimodal, multi-agent framework that mirrors stepwise clinical reasoning. It decomposes the task into Retrieval, Draft, Refiner, Vision, and Synthesis agents, and combines retrieval-augmented generation with cross-modal grounding to improve factuality and interpretability. Empirical results on IU X-ray show notable gains over a strong single-agent baseline across automatic metrics and LLM-based judgments, with case studies highlighting enhanced factuality and structured reporting. The modular, clinically aligned design offers a pathway to more trustworthy, explainable AI support in radiology and other multimodal medical tasks.

Abstract

Radiology report generation (RRG) aims to automatically produce diagnostic reports from medical images, with the potential to enhance clinical workflows and reduce radiologists' workload. While recent approaches leveraging multimodal large language models (MLLMs) and retrieval-augmented generation (RAG) have achieved strong results, they continue to face challenges such as factual inconsistency, hallucination, and cross-modal misalignment. We propose a multimodal multi-agent framework for RRG that aligns with the stepwise clinical reasoning workflow, where task-specific agents handle retrieval, draft generation, visual analysis, refinement, and synthesis. Experimental results demonstrate that our approach outperforms a strong baseline in both automatic metrics and LLM-based evaluations, producing more accurate, structured, and interpretable reports. This work highlights the potential of clinically aligned multi-agent frameworks to support explainable and trustworthy clinical AI applications.

A Multimodal Multi-Agent Framework for Radiology Report Generation

TL;DR

This work tackles radiology report generation (RRG) by introducing a multimodal, multi-agent framework that mirrors stepwise clinical reasoning. It decomposes the task into Retrieval, Draft, Refiner, Vision, and Synthesis agents, and combines retrieval-augmented generation with cross-modal grounding to improve factuality and interpretability. Empirical results on IU X-ray show notable gains over a strong single-agent baseline across automatic metrics and LLM-based judgments, with case studies highlighting enhanced factuality and structured reporting. The modular, clinically aligned design offers a pathway to more trustworthy, explainable AI support in radiology and other multimodal medical tasks.

Abstract

Radiology report generation (RRG) aims to automatically produce diagnostic reports from medical images, with the potential to enhance clinical workflows and reduce radiologists' workload. While recent approaches leveraging multimodal large language models (MLLMs) and retrieval-augmented generation (RAG) have achieved strong results, they continue to face challenges such as factual inconsistency, hallucination, and cross-modal misalignment. We propose a multimodal multi-agent framework for RRG that aligns with the stepwise clinical reasoning workflow, where task-specific agents handle retrieval, draft generation, visual analysis, refinement, and synthesis. Experimental results demonstrate that our approach outperforms a strong baseline in both automatic metrics and LLM-based evaluations, producing more accurate, structured, and interpretable reports. This work highlights the potential of clinically aligned multi-agent frameworks to support explainable and trustworthy clinical AI applications.
Paper Structure (15 sections, 2 figures, 2 tables)

This paper contains 15 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Retrieval Agent
  5. Draft Agent
  6. Refiner Agent
  7. Vision Agent
  8. Synthesis Agent
  9. Experiments
  10. Experimental Setup
  11. Results
  12. Quantitative Analysis
  13. Qualitative Analysis
  14. Discussion
  15. Conclusion

Figures (2)

  • Figure 1: Overview of our proposed multi-agent framework for automated RRG. The system decomposes the task into five agents: (1) Retrieval Agent selects top-k similar reports. (2) Draft Agent generates a preliminary report from the retrieved texts. (3) Refiner Agent distills key clinical findings. (4) Vision Agent generates a visual description of the image. (5) Synthesis Agent integrates these outputs to produce the final report.
  • Figure 2: A case study showing that retrieval and key findings help overcome the limitations of a vision-only agent.