Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusion-Augmented Large Language Models: Boosting Diagnostic Trustworthiness via Model Consensus

Md Kamrul Siam, Md Jobair Hossain Faruk, Jerry Q. Cheng, Huanying Gu

TL;DR

The paper tackles the reliability of large language models in radiology by introducing a model-agnostic consensus framework that coordinates two LLMs via structured prompts and semantic agreement. It integrates multimodal inputs by generating synthetic clinical notes and encoding images for access by LLMs, validated on the CheXpert dataset. Key contributions include a modular fusion architecture, an interpretable consensus metric based on semantic similarity, and empirical evidence that model agreement yields higher diagnostic accuracy than any single model in both unimodal and multimodal settings. The work demonstrates a practical path to safer AI-assisted radiology with minimal computational overhead, offering a fallback mechanism for uncertain cases and potential reductions in diagnostic errors.

Abstract

This study presents a novel multi-model fusion framework leveraging two state-of-the-art large language models (LLMs), ChatGPT and Claude, to enhance the reliability of chest X-ray interpretation on the CheXpert dataset. From the full CheXpert corpus of 224,316 chest radiographs, we randomly selected 234 radiologist-annotated studies to evaluate unimodal performance using image-only prompts. In this setting, ChatGPT and Claude achieved diagnostic accuracies of 62.8% and 76.9%, respectively. A similarity-based consensus approach, using a 95% output similarity threshold, improved accuracy to 77.6%. To assess the impact of multimodal inputs, we then generated synthetic clinical notes following the MIMIC-CXR template and evaluated a separate subset of 50 randomly selected cases paired with both images and synthetic text. On this multimodal cohort, performance improved to 84% for ChatGPT and 76% for Claude, while consensus accuracy reached 91.3%. Across both experimental conditions, agreement-based fusion consistently outperformed individual models. These findings highlight the utility of integrating complementary modalities and using output-level consensus to improve the trustworthiness and clinical utility of AI-assisted radiological diagnosis, offering a practical path to reduce diagnostic errors with minimal computational overhead.

Fusion-Augmented Large Language Models: Boosting Diagnostic Trustworthiness via Model Consensus

TL;DR

The paper tackles the reliability of large language models in radiology by introducing a model-agnostic consensus framework that coordinates two LLMs via structured prompts and semantic agreement. It integrates multimodal inputs by generating synthetic clinical notes and encoding images for access by LLMs, validated on the CheXpert dataset. Key contributions include a modular fusion architecture, an interpretable consensus metric based on semantic similarity, and empirical evidence that model agreement yields higher diagnostic accuracy than any single model in both unimodal and multimodal settings. The work demonstrates a practical path to safer AI-assisted radiology with minimal computational overhead, offering a fallback mechanism for uncertain cases and potential reductions in diagnostic errors.

Abstract

This study presents a novel multi-model fusion framework leveraging two state-of-the-art large language models (LLMs), ChatGPT and Claude, to enhance the reliability of chest X-ray interpretation on the CheXpert dataset. From the full CheXpert corpus of 224,316 chest radiographs, we randomly selected 234 radiologist-annotated studies to evaluate unimodal performance using image-only prompts. In this setting, ChatGPT and Claude achieved diagnostic accuracies of 62.8% and 76.9%, respectively. A similarity-based consensus approach, using a 95% output similarity threshold, improved accuracy to 77.6%. To assess the impact of multimodal inputs, we then generated synthetic clinical notes following the MIMIC-CXR template and evaluated a separate subset of 50 randomly selected cases paired with both images and synthetic text. On this multimodal cohort, performance improved to 84% for ChatGPT and 76% for Claude, while consensus accuracy reached 91.3%. Across both experimental conditions, agreement-based fusion consistently outperformed individual models. These findings highlight the utility of integrating complementary modalities and using output-level consensus to improve the trustworthiness and clinical utility of AI-assisted radiological diagnosis, offering a practical path to reduce diagnostic errors with minimal computational overhead.
Paper Structure (21 sections, 6 equations, 3 figures, 9 tables)

This paper contains 21 sections, 6 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in Medical Decision-Making
  4. Ensemble Methods in AI
  5. Multimodal Fusion in Medical AI
  6. Methodology
  7. Synthetic Clinical Note Generation
  8. Model Pipeline and Fusion Framework
  9. Statistical Analysis
  10. Unimodal Analysis
  11. Multimodal Analysis
  12. Results
  13. Unimodal Performance: Image Only
  14. Multimodal Performance: Image + Clinical Notes
  15. Comparative Analysis on the 50-Case Subset
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Comparison between (a) a de-identified snippet from a MIMIC-CXR clinical report and (b) an example of synthetic note generated for our experiment.
  • Figure 2: CheXpert: predict the probability of 14 findings from the frontal and lateral chest radiographs (from Irvin et al.irvin2019chexpert).
  • Figure 3: Proposed AI-Driven Medical Diagnosis Workflow