AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation

TL;DR

AnatomiX introduces an anatomy‑aware grounded multimodal LLM for chest X‑ray interpretation that separates anatomical perception from downstream reasoning. The Anatomy Perception Module first detects and represents thoracic structures, then a MedGemma‑based LM, augmented with anatomy tokens and a vector‑retrieved textual context, performs nine radiology tasks within a unified prompt framework. The two‑stage design, augmented by a soft contrastive loss and a lightweight vector database, yields substantial gains in grounding (up to >25%), plus competitive or superior performance on report generation, image understanding, and VQA. This anatomy‑centric approach improves spatial reasoning in medical MLLMs and highlights the value of domain‑specific architectural design for reliable radiology AI assistance.

Abstract

Multimodal medical large language models have shown impressive progress in chest X-ray interpretation but continue to face challenges in spatial reasoning and anatomical understanding. Although existing grounding techniques improve overall performance, they often fail to establish a true anatomical correspondence, resulting in incorrect anatomical understanding in the medical domain. To address this gap, we introduce AnatomiX, a multitask multimodal large language model explicitly designed for anatomically grounded chest X-ray interpretation. Inspired by the radiological workflow, AnatomiX adopts a two stage approach: first, it identifies anatomical structures and extracts their features, and then leverages a large language model to perform diverse downstream tasks such as phrase grounding, report generation, visual question answering, and image understanding. Extensive experiments across multiple benchmarks demonstrate that AnatomiX achieves superior anatomical reasoning and delivers over 25% improvement in performance on anatomy grounding, phrase grounding, grounded diagnosis and grounded captioning tasks compared to existing approaches. Code and pretrained model are available at https://github.com/aneesurhashmi/anatomix

Figures (9)

• Figure 1: Comparison between AnatomiX and RadVLM deperrois2025radvlm in anatomy understanding. (a) and (b) show both models predicting the disease on the correct side (color scheme: red for model's output, green for all ground truth locations). (c), (d) and (e) show models' outputs for the same image flipped on the vertical axis (left $\leftrightarrow$ right), where RadVLM completely fails to recognize the correct anatomical object, while AnatomiX successfully recognizes the correct anatomies, showcasing high anatomical understanding
• Figure 2: Anatomy Perception Module (APM) architecture $\mathrm{\textbf{(a)}}$: The encoder $\mathcal{E}$ outputs image embedding $\mathrm{I}_p$, while the decoder $\mathcal{D}$ and feature extraction module $\mathcal{M}$ output object bounding boxes $\hat{y}_{box}$, and anatomical object tokens $\hat{\mathrm{O}}_A$, respectively. Different colors in $\Tilde{\mathrm{O}}$, $\mathrm{O}$ and $\hat{\mathrm{O}}_A$ represent specific anatomical objects. $\mathrm{\textbf{(b)}}$ shows the contrastive alignment using frozen sentence encoder $\mathcal{S}$ and self-similarity loss. $\mathrm{\textbf{(c)}}$: The vector database ($\mathcal{V}_{DB}$) contains the text sentences and embeddings used for contrastive retrieval. (Bottom right): APM uses $\mathrm{\textbf{(a)}}$ and $\mathrm{\textbf{(b)}}$ during training, and replaces $\mathrm{\textbf{(b)}}$ with $\mathrm{\textbf{(c)}}$ during inference. The $\mathcal{P}$ represent different FC projectors described in section \ref{['sec:stage_1']}.
• Figure 3: Overall architecture of AnatomiX. The outputs of the APM and the user prompt $U$ are added to a structured multimodal prompt template before being passed to the $\mathcal{LM},$ which generates the response $T$. $\mathcal{P}_{im}$ and $\mathcal{P}_{A}$ represent FC projectors as described in section \ref{['sec:sec_llm']}.
• Figure 4: Set of 9 radiology tasks (middle circle) spanning 4 categories (inner circle) done by AnatomiX and the datasets used (outer circle).
• Figure 5: Sample input–output pairs and comparison with second best models on grounding tasks. The upper panels show outputs from our model across four tasks. The lower panels compare AnatomiX with RadVLM deperrois2025radvlm for phrase and anatomy grounding, and with CheXagent chen2024chexagent for grounded diagnosis and captioning. Box colors: blue = user input, green = ground truth, red = model output.