Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting

Naman Sharma

TL;DR

The work addresses the challenge of reliable chest X-ray interpretation using vision-language models and the risks of confident hallucinations in radiology reports. It systematically evaluates state-of-the-art VLMs (e.g., CheXagent, BioViL-T, BLIP-2) and introduces an uncertainty-aware, agent-based reporting workflow that leverages a probed vision encoder and phrase grounding to generate findings with calibrated confidence. Key contributions include detailed linear probing of CheXagent components, a modular CXR-agent architecture that combines pathology detection, localisation, and LLM-driven report generation, and a data-collection platform for expert clinical evaluation. Findings show that vision encoders from foundational models generalize well across datasets and that LLM bottlenecks and data diversity limit end-to-end performance; the proposed uncertainty-aware approach improves interpretability and safety, though localization and data scarcity remain critical challenges. The work underscores the need for larger, diverse paired scan-report datasets and careful normal-vs-abnormal evaluation to enable safe, clinically useful AI radiology tools.

Abstract

Recently large vision-language models have shown potential when interpreting complex images and generating natural language descriptions using advanced reasoning. Medicine's inherently multimodal nature incorporating scans and text-based medical histories to write reports makes it conducive to benefit from these leaps in AI capabilities. We evaluate the publicly available, state of the art, foundational vision-language models for chest X-ray interpretation across several datasets and benchmarks. We use linear probes to evaluate the performance of various components including CheXagent's vision transformer and Q-former, which outperform the industry-standard Torch X-ray Vision models across many different datasets showing robust generalisation capabilities. Importantly, we find that vision-language models often hallucinate with confident language, which slows down clinical interpretation. Based on these findings, we develop an agent-based vision-language approach for report generation using CheXagent's linear probes and BioViL-T's phrase grounding tools to generate uncertainty-aware radiology reports with pathologies localised and described based on their likelihood. We thoroughly evaluate our vision-language agents using NLP metrics, chest X-ray benchmarks and clinical evaluations by developing an evaluation platform to perform a user study with respiratory specialists. Our results show considerable improvements in accuracy, interpretability and safety of the AI-generated reports. We stress the importance of analysing results for normal and abnormal scans separately. Finally, we emphasise the need for larger paired (scan and report) datasets alongside data augmentation to tackle overfitting seen in these large vision-language models.

CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting

TL;DR

The work addresses the challenge of reliable chest X-ray interpretation using vision-language models and the risks of confident hallucinations in radiology reports. It systematically evaluates state-of-the-art VLMs (e.g., CheXagent, BioViL-T, BLIP-2) and introduces an uncertainty-aware, agent-based reporting workflow that leverages a probed vision encoder and phrase grounding to generate findings with calibrated confidence. Key contributions include detailed linear probing of CheXagent components, a modular CXR-agent architecture that combines pathology detection, localisation, and LLM-driven report generation, and a data-collection platform for expert clinical evaluation. Findings show that vision encoders from foundational models generalize well across datasets and that LLM bottlenecks and data diversity limit end-to-end performance; the proposed uncertainty-aware approach improves interpretability and safety, though localization and data scarcity remain critical challenges. The work underscores the need for larger, diverse paired scan-report datasets and careful normal-vs-abnormal evaluation to enable safe, clinically useful AI radiology tools.

Abstract

Recently large vision-language models have shown potential when interpreting complex images and generating natural language descriptions using advanced reasoning. Medicine's inherently multimodal nature incorporating scans and text-based medical histories to write reports makes it conducive to benefit from these leaps in AI capabilities. We evaluate the publicly available, state of the art, foundational vision-language models for chest X-ray interpretation across several datasets and benchmarks. We use linear probes to evaluate the performance of various components including CheXagent's vision transformer and Q-former, which outperform the industry-standard Torch X-ray Vision models across many different datasets showing robust generalisation capabilities. Importantly, we find that vision-language models often hallucinate with confident language, which slows down clinical interpretation. Based on these findings, we develop an agent-based vision-language approach for report generation using CheXagent's linear probes and BioViL-T's phrase grounding tools to generate uncertainty-aware radiology reports with pathologies localised and described based on their likelihood. We thoroughly evaluate our vision-language agents using NLP metrics, chest X-ray benchmarks and clinical evaluations by developing an evaluation platform to perform a user study with respiratory specialists. Our results show considerable improvements in accuracy, interpretability and safety of the AI-generated reports. We stress the importance of analysing results for normal and abnormal scans separately. Finally, we emphasise the need for larger paired (scan and report) datasets alongside data augmentation to tackle overfitting seen in these large vision-language models.
Paper Structure (75 sections, 9 equations, 10 figures, 1 table)

This paper contains 75 sections, 9 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Objectives
  3. Novel Contributions
  4. Ethical Issues
  5. Background
  6. Medical context
  7. Chest X-rays (CXR) and radiologist reports
  8. CXR datasets
  9. Convolutional Neural Networks (CNNs)
  10. ResNets
  11. Transformers
  12. Limitations of RNNs
  13. Self-attention mechanism
  14. Multi-head attention
  15. Transformer Architecture
  16. ...and 60 more sections

Figures (10)

  • Figure 1: The four-stage training process for CheXagent
  • Figure 2: Google's rubric to compare two radiology reports to be used by certified radiologists
  • Figure 3: An example training pair for the fine-grained image understanding training task
  • Figure 4: Comparison of pathology classification results across different prompts and VinDr/CheXpert dataset
  • Figure 5: Comparison of pathology classification between CheXagent, CheXagent probes and Torch X-ray vision models (*: best CheXagent configuration from figure \ref{['fig:exact_match_CheXagent']} (pathologies prompt at temperature 0.5))
  • ...and 5 more figures