Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment

TL;DR

Malenia tackles zero-shot 3D lesion segmentation by bridging image-level vision-language pre-training and pixel-level prediction through multi-scale mask-attribute alignment and a Cross-Modal Knowledge Injection module. It represents unseen lesions by linking their visual appearance to a set of eight structured attribute descriptions extracted from reports, and it fuses masked visual features with attribute text to guide segmentation. Across MSD, KiTS23, and in-house data, Malenia achieves state-of-the-art zero-shot performance and strong results on seen categories, with ablations confirming the contribution of multi-scale alignment, multi-positive contrastive learning, and CMKI. The approach reduces reliance on manual prompts and offers a practical, prompt-free, cross-modal segmentation framework for clinical deployment.

Abstract

Recent advancements in medical vision-language pre-training models have driven significant progress in zero-shot disease recognition. However, transferring image-level knowledge to pixel-level tasks, such as lesion segmentation in 3D CT scans, remains a critical challenge. Due to the complexity and variability of pathological visual characteristics, existing methods struggle to align fine-grained lesion features not encountered during training with disease-related textual representations. In this paper, we present Malenia, a novel multi-scale lesion-level mask-attribute alignment framework, specifically designed for 3D zero-shot lesion segmentation. Malenia improves the compatibility between mask representations and their associated elemental attributes, explicitly linking the visual features of unseen lesions with the extensible knowledge learned from previously seen ones. Furthermore, we design a Cross-Modal Knowledge Injection module to enhance both visual and textual features with mutually beneficial information, effectively guiding the generation of segmentation results. Comprehensive experiments across three datasets and 12 lesion categories validate the superior performance of Malenia.

Figures (8)

• Figure 1: Overview of Malenia. The key contributions of our work are two simple but effective designs: the Multi-scale Fine-Grained Mask-Attribute Alignment and the CMKI module.
• Figure 2: Overview of the inference process of Malenia. (1) Step-I: Image Partitioning via Mask Tokens. Test CT images are divided into regions, each represented by mask tokens. (2) Step-II: Mask-attribute matching. Each mask token is associated with stored attribute embeddings. (3) Step-III: Cross-modal fusion and mask prediction. Information from mask tokens and text embeddings is fused to generate segmentation masks. (4) Step-IV: Disease identification via attribute-querying. The Clinical Knowledge Table links the predicted attributes to specific disease categories for precise diagnosis.
• Figure 3: Qualitative visualizations of Malenia and other competing methods on both unseen and seen lesions. The segmentation results, presented from top to bottom and left to right, include Hepatic Vessel Tumor, Pancreas Cyst, Colon Tumor, and Lung Tumor.
• Figure 4: Ablation studies on segmentation performance of unseen lesions using different numbers of mask tokens.
• Figure 5: Ablation studies on segmentation performance of unseen lesions using different numbers of attribute aspects.