XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography

TL;DR

XBench addresses the critical need for reliable grounding in medical vision-language models by introducing a unified benchmark for chest radiographs that jointly evaluates recognition and cross-modal grounding. It analyzes seven CLIP-style VLMs across 36 findings and 12,601 cases, using both radiologist-annotated regions and multiple grounding metrics to reveal how domain-specific pretraining improves localization for large lesions while small, diffuse findings remain challenging. A strong coupling between recognition performance and grounding is observed (e.g., $R^2=0.92$), yet significant per-pathology gaps and calibration issues persist, underscoring the need for size-aware interpretability benchmarks before clinical deployment. The XBench framework, with its Dataset/Model/Metrics modular design, provides a rigorous foundation for evaluating and improving medically oriented multimodal explanations and sets a baseline for future domain-adapted models.

Abstract

Vision-language models (VLMs) have recently shown remarkable zero-shot performance in medical image understanding, yet their grounding ability, the extent to which textual concepts align with visual evidence, remains underexplored. In the medical domain, however, reliable grounding is essential for interpretability and clinical adoption. In this work, we present the first systematic benchmark for evaluating cross-modal interpretability in chest X-rays across seven CLIP-style VLM variants. We generate visual explanations using cross-attention and similarity-based localization maps, and quantitatively assess their alignment with radiologist-annotated regions across multiple pathologies. Our analysis reveals that: (1) while all VLM variants demonstrate reasonable localization for large and well-defined pathologies, their performance substantially degrades for small or diffuse lesions; (2) models that are pretrained on chest X-ray-specific datasets exhibit improved alignment compared to those trained on general-domain data. (3) The overall recognition ability and grounding ability of the model are strongly correlated. These findings underscore that current VLMs, despite their strong recognition ability, still fall short in clinically reliable grounding, highlighting the need for targeted interpretability benchmarks before deployment in medical practice. XBench code is available at https://github.com/Roypic/Benchmarkingattention

Figures (4)

• Figure 1: Overview of the unified evaluation framework for medical vision-language models. The framework integrates three main components: Dataset, Model, and Metrics. The Dataset Wrapper organizes multi-source datasets (36 diseases, 12,601 cases) including images, annotations, prompt templates, and augmentations from RSNA Pneumoniarsna, Covid19-ruraldesai2020chest, ChestDet-10liu2020chestx, SIIMsiim-acr-pneumothorax-segmentation, CheXlocalizesaporta2022benchmarking, ChestXray14 Detectionwang2017chestx, and Vindr-CXRnguyen2022vindr. The Model Wrapper standardizes inference logic across vision-language models, encapsulating the image encoder, text encoder, and fusion module; it supports representative models such as CLIP, BiomedCLIP, MedKILP, KAD, MAVL, CARZero, and DeViDe. The Metrics module unifies both Grounding Metrics (e.g., Pointing Game, Dice, IoU with fixed or searched thresholds) and Classification Metrics (e.g., AUC, Accuracy, F1), enabling comprehensive and comparable evaluation across datasets and models.
• Figure 2: Visual comparison of disease localization across six vision-language models on chest X-rays. Blue boxes mark ground-truth regions. CARZero and DeViDe show more accurate and focused attention for Pneumothorax, Consolidation, and Lung Opacity.
• Figure 3: Correlation between disease classification and grounding accuracy across vision-language models.
• Figure 4: Comparison of Dice coefficients under fixed (0.5) and optimized thresholds across seven vision-language models. CARZero achieves the highest Dice scores in both settings, indicating superior lesion localization consistency.