S-Chain: Structured Visual Chain-of-Thought For Medicine

TL;DR

S-Chain introduces a large-scale, expert-annotated dataset that tightly links structured visual reasoning steps to specific image regions in medical imaging, addressing the core challenge of faithful interpretation in medical VLMs. By organizing reasoning into four stages—localization, description, grading, and classification—the dataset enables autoregressive SV-CoT supervision and substantially improves grounding fidelity, interpretability, and robustness over GPT-based synthetic approaches across both medical and general VLMs. The work further demonstrates that combining SV-CoT with retrieval-augmented generation (MedRAG/MIRIAD) yields additional gains, while proposing regularization mechanisms to better align visual evidence with reasoning. Overall, S-Chain establishes a new benchmark and methodological framework for trustworthy, visually grounded medical reasoning with practical implications for safer clinical AI deployments.

Abstract

Faithful reasoning in medical vision-language models (VLMs) requires not only accurate predictions but also transparent alignment between textual rationales and visual evidence. While Chain-of-Thought (CoT) prompting has shown promise in medical visual question answering (VQA), no large-scale expert-level dataset has captured stepwise reasoning with precise visual grounding. We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT), explicitly linking visual regions to reasoning steps. The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability. Using S-Chain, we benchmark state-of-the-art medical VLMs (ExGra-Med, LLaVA-Med) and general-purpose VLMs (Qwen2.5-VL, InternVL2.5), showing that SV-CoT supervision significantly improves interpretability, grounding fidelity, and robustness. Beyond benchmarking, we study its synergy with retrieval-augmented generation, revealing how domain knowledge and visual grounding interact during autoregressive reasoning. Finally, we propose a new mechanism that strengthens the alignment between visual evidence and reasoning, improving both reliability and efficiency. S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical VLMs.

Figures (18)

• Figure 1: Overview of the S-Chain dataset with annotations. Each image is paired with (Q1) localization via bounding boxes, (Q2) lesion descriptions, and (Q3) lesion grading using standardized scales (e.g., Koedam, GCA, MTA). These stepwise annotations ground reasoning in visual evidence, enabling interpretable and reliable medical .
• Figure 2: Annotation pipeline. Experts first select representative 2D slices from volumes (1), then localize with bounding boxes (2). Abnormalities are described through structured reasoning notes (3) and graded using standardized visual rating scales (4). Annotations undergo expert consensus for quality control (5), and finally, all reasoning steps are translated into several languages with expert validation (6), yielding a multilingual, expert-grounded dataset. (See Appendix Section \ref{['sec:dataset_examples']} for some dataset examples, e.g. Figure \ref{['fig:datasetexamples_english_nondementia']}).
• Figure 3: Accuracy of medicaltrained with the base setting (Q4-only), synthetic GPT-4.1 , and expert-annotated S-Chain (ours). S-Chain consistently improves performance across models, with closed-source APIs (GPT-4.1, Grok-4, Gemini-2.5-Flash) shown for 8-shot reference.
• Figure 4: Accuracy of general-purposetrained with the base setting (Q4-only), synthetic GPT-4.1 , and expert-annotated S-Chain (ours). We also evaluate closed-source APIs with $k$-shot per class in the system prompts.
• Figure 5: A query to MIRIAD for the retrieval of the top relevant descriptions.