A Medical Multimodal Diagnostic Framework Integrating Vision-Language Models and Logic Tree Reasoning

TL;DR

The paper addresses the challenge that combining clinical text and medical images with large models can lead to hallucinations and opaque reasoning in multimodal medical AI. It presents a logic-regularized framework built on LLaVA that integrates vision–language grounding with a reasoning controller and a logic-tree generator, trained end-to-end using Dynamic Advantage Policy Optimization (DAPO). Key contributions include improved diagnostic accuracy and more auditable reasoning traces on benchmarks like MedXpertQA, VQA-RAD, PathVQA, and PubMedQA, while preserving performance on text-only tasks. This work advances trustworthy multimodal medical AI by delivering verifiable, stepwise explanations that clinicians can inspect, potentially enabling safer real-world deployment and guiding future enhancements such as retrieval-augmented reasoning and interpretable visualization of reasoning structures.

Abstract

With the rapid growth of large language models (LLMs) and vision-language models (VLMs) in medicine, simply integrating clinical text and medical imaging does not guarantee reliable reasoning. Existing multimodal models often produce hallucinations or inconsistent chains of thought, limiting clinical trust. We propose a diagnostic framework built upon LLaVA that combines vision-language alignment with logic-regularized reasoning. The system includes an input encoder for text and images, a projection module for cross-modal alignment, a reasoning controller that decomposes diagnostic tasks into steps, and a logic tree generator that assembles stepwise premises into verifiable conclusions. Evaluations on MedXpertQA and other benchmarks show that our method improves diagnostic accuracy and yields more interpretable reasoning traces on multimodal tasks, while remaining competitive on text-only settings. These results suggest a promising step toward trustworthy multimodal medical AI.

Figures (2)

• Figure 1: Overview. Left: baseline VLMs rely on implicit CoT, causing hallucinations, inconsistency, and weak grounding. Right: our framework integrates vision--language alignment and logic-regularized reasoning, yielding traceable logic trees and consistent diagnoses.
• Figure 2: Multimodal logic-regularized reasoning framework. Patient history, clinical notes, and medical images (e.g., CT, MRI, X-ray) are encoded by text and vision encoders. Their embeddings are fed into a large language model with a DAPO policy optimizer, producing multiple candidate reasoning rollouts. Each rollout is parsed by the logic tree extractor into syllogistic premises and conclusions, forming verifiable logic trees. The final diagnosis is obtained together with a traceable reasoning chain, improving both accuracy and interpretability.