Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Models

Muhammad Imran, Chi Lee, Yugyung Lee

TL;DR

MATEX addresses the critical need for interpretable medical vision-language models by uniting multi-scale attention rollout, text-guided spatial priors, and cross-layer consistency to generate anatomically grounded attribution maps. The framework fuses gradient signals, hierarchical attention, and language-derived anatomical priors to yield clinically meaningful explanations that align with radiologist reasoning. Evaluated on the MS-CXR dataset, MATEX outperforms state-of-the-art baselines across image and text attribution metrics, demonstrating sharper localization and robust, diagnosis-aligned guidance. This work advances trust and transparency in radiology AI by coupling architectural signals with linguistic context, enabling more reliable human-AI collaboration in high-stakes imaging tasks.

Abstract

We introduce MATEX (Multi-scale Attention and Text-guided Explainability), a novel framework that advances interpretability in medical vision-language models by incorporating anatomically informed spatial reasoning. MATEX synergistically combines multi-layer attention rollout, text-guided spatial priors, and layer consistency analysis to produce precise, stable, and clinically meaningful gradient attribution maps. By addressing key limitations of prior methods, such as spatial imprecision, lack of anatomical grounding, and limited attention granularity, MATEX enables more faithful and interpretable model explanations. Evaluated on the MS-CXR dataset, MATEX outperforms the state-of-the-art M2IB approach in both spatial precision and alignment with expert-annotated findings. These results highlight MATEX's potential to enhance trust and transparency in radiological AI applications.

MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Models

TL;DR

MATEX addresses the critical need for interpretable medical vision-language models by uniting multi-scale attention rollout, text-guided spatial priors, and cross-layer consistency to generate anatomically grounded attribution maps. The framework fuses gradient signals, hierarchical attention, and language-derived anatomical priors to yield clinically meaningful explanations that align with radiologist reasoning. Evaluated on the MS-CXR dataset, MATEX outperforms state-of-the-art baselines across image and text attribution metrics, demonstrating sharper localization and robust, diagnosis-aligned guidance. This work advances trust and transparency in radiology AI by coupling architectural signals with linguistic context, enabling more reliable human-AI collaboration in high-stakes imaging tasks.

Abstract

We introduce MATEX (Multi-scale Attention and Text-guided Explainability), a novel framework that advances interpretability in medical vision-language models by incorporating anatomically informed spatial reasoning. MATEX synergistically combines multi-layer attention rollout, text-guided spatial priors, and layer consistency analysis to produce precise, stable, and clinically meaningful gradient attribution maps. By addressing key limitations of prior methods, such as spatial imprecision, lack of anatomical grounding, and limited attention granularity, MATEX enables more faithful and interpretable model explanations. Evaluated on the MS-CXR dataset, MATEX outperforms the state-of-the-art M2IB approach in both spatial precision and alignment with expert-annotated findings. These results highlight MATEX's potential to enhance trust and transparency in radiological AI applications.
Paper Structure (16 sections, 7 equations, 3 figures, 2 tables)

This paper contains 16 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. MATEX Framework Overview
  5. Multi-Scale Attention Rollout and Consistency
  6. Text-Guided Spatial Priors
  7. Multi-Component Attribution Fusion
  8. Experiments and Results
  9. Dataset and Implementation Details
  10. Evaluation Metrics
  11. Quantitative Results
  12. Findings.
  13. Qualitative Evaluation
  14. Findings.
  15. Discussion and Limitations
  16. ...and 1 more sections

Figures (3)

  • Figure 1: MATEX Text-aware and anatomically grounded explanations produced by MATEX. Heatmaps localize lung cancer regions in the input image, while highlighted caption keywords indicate the clinical concepts driving the prediction.
  • Figure 2: MATEX architecture.Top: the CLIP image encoder embeds a chest X-ray and computes value magnitudes, raw attention, and weighted spatial attention; a similarity score $S(\mathbf{f}_i,\mathbf{f}_r)$ provides gradients $\partial S/\partial \mathbf{v}_{\text{base}}$ that drive the visual relevance module (VR_I) via multi-scale enhancement to produce a spatial explanation map. Bottom: the CLIP text encoder embeds the input report phrase (e.g., "local consolidation in left mid lung") and applies token attention (scaled by $\sqrt{d_k}$) to obtain token relevance (TR_I) through multi-scale processing, yielding token-level attributions and anatomical cues. The final explanation combines VR_I and TR_I to align salient image regions with clinically meaningful tokens. Here $d_k$ is the attention key/query dimension, $C$ is the visual channel dimension, and $m$ is the number of attention heads.
  • Figure 3: Qualitative comparison between MATEX and M2IB on MS-CXR. Each column shows a clinical case with MATEX (Top) and M2IB (Bottom). Left to Right: (1) Left upper lobe consolidation: MATEX localizes sharply; M2IB is diffuse. (2) Bilateral lower lobe consolidation: MATEX captures both lobes; M2IB is less focused. (3) Right mid-to-upper zone opacity: MATEX aligns closely with pathology. (4) Left lower lobe opacification: MATEX offers better boundary definition. (5) Left mid-lung consolidation: MATEX maintains anatomical fidelity. Overall, MATEX shows superior spatial precision and clinical alignment.