M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding

TL;DR

The paper introduces M3CoTBench, a benchmark to evaluate chain-of-thought reasoning in medical multimodal language models for image understanding. It features a 1,079-image, 24-modality dataset spanning 13 tasks, with CoT-specific metrics assessing correctness, efficiency, impact, and consistency, plus a path consistency measure. The curation pipeline combines automated data generation with expert and multi-model calibration to ensure clinically meaningful CoT annotations. Across diverse MLLMs, the study reveals that CoT can both help and hinder medical reasoning, underscoring the need for interpretable, clinically grounded CoT and providing a foundation for future improvements in trustworthy healthcare AI.

Abstract

Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models by encouraging step-by-step intermediate reasoning, and recent advances have extended this paradigm to Multimodal Large Language Models (MLLMs). In the medical domain, where diagnostic decisions depend on nuanced visual cues and sequential reasoning, CoT aligns naturally with clinical thinking processes. However, Current benchmarks for medical image understanding generally focus on the final answer while ignoring the reasoning path. An opaque process lacks reliable bases for judgment, making it difficult to assist doctors in diagnosis. To address this gap, we introduce a new M3CoTBench benchmark specifically designed to evaluate the correctness, efficiency, impact, and consistency of CoT reasoning in medical image understanding. M3CoTBench features 1) a diverse, multi-level difficulty dataset covering 24 examination types, 2) 13 varying-difficulty tasks, 3) a suite of CoT-specific evaluation metrics (correctness, efficiency, impact, and consistency) tailored to clinical reasoning, and 4) a performance analysis of multiple MLLMs. M3CoTBench systematically evaluates CoT reasoning across diverse medical imaging tasks, revealing current limitations of MLLMs in generating reliable and clinically interpretable reasoning, and aims to foster the development of transparent, trustworthy, and diagnostically accurate AI systems for healthcare. Project page at https://juntaojianggavin.github.io/projects/M3CoTBench/.

Figures (8)

• Figure 1: Curation of M3CoTBench benchmark that encompasses three sections: 1) carefully curated medical images from various public sources, 2) multi-type and multi-difficulty QA generation via LLMs and expert calibration, and 3) structured annotation of key reasoning steps aligned with clinical diagnostic workflows.
• Figure 2: Overview of M3CoTBench benchmark. Top: The benchmark covers 24 imaging modalities/examination types, 4 question types, and 13 clinical reasoning tasks. Middle: CoT annotation examples and 4 evaluation dimensions. Bottom: The distribution of image-QA pairs across a) modalities, b) question types, and c) tasks.
• Figure 3: Example image-question pairs for 13 tasks in M3CoTBench, including identifying examination types, image quality assessment, recognition, referring recognition, counting, localization, diagnosis, grading, symptom identification, clinical action planning, prediction, functional understanding, and causal reasoning
• Figure A1: Word cloud for abnormality and diseases included in M3CoTBench The word cloud below visualizes the frequency and variety of these labels, highlighting the spectrum of diagnostic conclusions and imaging findings represented.
• Figure A2: Image resolution distribution in M3CoTBench. Most images are concentrated below a width of 1200 and a height of 1500, though some exhibit higher resolutions.