Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation

TL;DR

This work tackles the challenge of annotation-heavy, ground-truth–dependent vision-grounded medical report generation by introducing Self-Supervised Anatomical Consistency Learning (SS-ACL). SS-ACL leverages a four-level hierarchical anatomy graph, recursive leaf-to-parent query restoration, and hierarchical masked reconstruction to align image regions with textual prompts, while region-level and global contrastive learning (L_ac, L_cl) encourage semantic consistency across images. A key contribution is anatomy-guided report generation via an anatomy-pathology prefix, enabling interpretable visual grounding without expert annotations and achieving strong performance on report quality (NLG) and clinical efficacy, as well as competitive zero-shot grounding on downstream vision tasks. The approach demonstrates robust generalization across datasets and tasks, offering a practical path toward annotation-free, interpretable multimodal medical reasoning and decision support.

Abstract

Vision-grounded medical report generation aims to produce clinically accurate descriptions of medical images, anchored in explicit visual evidence to improve interpretability and facilitate integration into clinical workflows. However, existing methods often rely on separately trained detection modules that require extensive expert annotations, introducing high labeling costs and limiting generalizability due to pathology distribution bias across datasets. To address these challenges, we propose Self-Supervised Anatomical Consistency Learning (SS-ACL) -- a novel and annotation-free framework that aligns generated reports with corresponding anatomical regions using simple textual prompts. SS-ACL constructs a hierarchical anatomical graph inspired by the invariant top-down inclusion structure of human anatomy, organizing entities by spatial location. It recursively reconstructs fine-grained anatomical regions to enforce intra-sample spatial alignment, inherently guiding attention maps toward visually relevant areas prompted by text. To further enhance inter-sample semantic alignment for abnormality recognition, SS-ACL introduces a region-level contrastive learning based on anatomical consistency. These aligned embeddings serve as priors for report generation, enabling attention maps to provide interpretable visual evidence. Extensive experiments demonstrate that SS-ACL, without relying on expert annotations, (i) generates accurate and visually grounded reports -- outperforming state-of-the-art methods by 10\% in lexical accuracy and 25\% in clinical efficacy, and (ii) achieves competitive performance on various downstream visual tasks, surpassing current leading visual foundation models by 8\% in zero-shot visual grounding.

Figures (8)

• Figure 1: Overview of the proposed SS-ACL. Left: the self-supervised anatomical consistency learning for annotation-free and grounded visual encoding. Right: the subsequent report generation under the guidance of generated anatomical representations.
• Figure 2: The proposed hierarchical anatomy graph along with the mapping between pathologies and anatomies.
• Figure 3: Qualitative phrase-grounding results. Given description phrases, we visualize the association of vision and language using the DINO attention map on the MS-CXR dataset. Ground-truth annotations are represented with dashed boxes. $\dagger$ denotes models pretrained using our training split.
• Figure 4: Qualitative report samples generated by SS-ACL with corresponding visual evidence and conventional SOTA models from the MIMIC-CXR test set. The green highlights indicate detected positive abnormalities, green underlines mark undetected positives, yellow highlights denote uncertain pathologies, red marks indicate incorrect estimations, blue marks signify negatives, and gray highlights represent unrelated descriptions.
• Figure 5: Comparison of inherent patch-wise attention maps generated by SS-ACL during training convergence. Rows correspond to query values in the Vision Transformer, while columns represent key values.