CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays

TL;DR

CXReasonBench addresses the gap in evaluating clinical diagnostic reasoning by measuring intermediate, structurally grounded steps rather than only final predictions. It builds on CheXStruct, an automated pipeline that extracts anatomy-based segmentation, landmarks, measurements, indices, and thresholds with rigorous QC from chest X-rays, enabling a multi-stage reference for evaluation. The benchmark employs two evaluation paths (direct reasoning and guided reasoning) across 18,988 QA pairs from 12 tasks on 1,200 MIMIC-CXR-JPG cases, revealing that even top LVLMs struggle with clinically valid, grounded reasoning and generalization. The work underscores the need for supervision signals that align visual grounding with diagnostic criteria and offers a scalable framework for transparent, interpretable assessment with potential extensions to report generation and multi-modal clinical reasoning.

Abstract

Recent progress in Large Vision-Language Models (LVLMs) has enabled promising applications in medical tasks, such as report generation and visual question answering. However, existing benchmarks focus mainly on the final diagnostic answer, offering limited insight into whether models engage in clinically meaningful reasoning. To address this, we present CheXStruct and CXReasonBench, a structured pipeline and benchmark built on the publicly available MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of intermediate reasoning steps directly from chest X-rays, such as segmenting anatomical regions, deriving anatomical landmarks and diagnostic measurements, computing diagnostic indices, and applying clinical thresholds. CXReasonBench leverages this pipeline to evaluate whether models can perform clinically valid reasoning steps and to what extent they can learn from structured guidance, enabling fine-grained and transparent assessment of diagnostic reasoning. The benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases, each paired with up to 4 visual inputs, and supports multi-path, multi-stage evaluation including visual grounding via anatomical region selection and diagnostic measurements. Even the strongest of 12 evaluated LVLMs struggle with structured reasoning and generalization, often failing to link abstract knowledge with anatomically grounded visual interpretation. The code is available at https://github.com/ttumyche/CXReasonBench

Figures (25)

• Figure 1: Overview of the CheXStruct pipeline. Given a chest X-ray, CheXStruct performs anatomical segmentation, derives anatomical landmarks and diagnostic measurements, computes diagnostic indices, and applies clinical thresholds, followed by task-specific quality control.
• Figure 2: Overview of the CXReasonBench evaluation pipeline. The evaluation begins with a direct diagnostic question, followed by two possible paths depending on the model’s response. Path 1 evaluates the model’s ability to reconstruct its reasoning through intermediate steps or to apply expert-defined criteria when necessary. Path 2 provides structured guidance to develop the reasoning process when the model expresses lack of confidence.
• Figure 3: Visualization of the anatomical structures of CXAS used in CheXStruct.
• Figure 4: Measurement of cardiomegaly using the cardiothoracic ratio (CTR).
• Figure 5: Measurement of the carina angle at the tracheal bifurcation.