Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation

Abstract

Universal medical image segmentation seeks to use a single foundational model to handle diverse tasks across multiple imaging modalities. However, existing approaches often rely heavily on manual visual prompts or retrieved reference images, which limits their automation and robustness. In addition, naive joint training across modalities often fails to address large domain shifts. To address these limitations, we propose Concept-to-Pixel (C2P), a novel prompt-free universal segmentation framework. C2P explicitly separates anatomical knowledge into two components: Geometric and Semantic representations. It leverages Multimodal Large Language Models (MLLMs) to distill abstract, high-level medical concepts into learnable Semantic Tokens and introduces explicitly supervised Geometric Tokens to enforce universal physical and structural constraints. These disentangled tokens interact deeply with image features to generate input-specific dynamic kernels for precise mask prediction. Furthermore, we introduce a Geometry-Aware Inference Consensus mechanism, which utilizes the model's predicted geometric constraints to assess prediction reliability and suppress outliers. Extensive experiments and analysis on a unified benchmark comprising eight diverse datasets across seven modalities demonstrate the significant superiority of our jointly trained approach, compared to universe- or single-model approaches. Remarkably, our unified model demonstrates strong generalization, achieving impressive results not only on zero-shot tasks involving unseen cases but also in cross-modal transfers across similar tasks. Code is available at: https://github.com/Yundi218/Concept-to-Pixel

Figures (11)

• Figure 1: Comparison of existing medical image segmentation paradigms and our proposed Concept-to-Pixel framework. Unlike existing paradigms that suffer from limited versatility (A), limited manual prompts (B), or reliance on reference quality (C), our prompt-free model (D), inspired by clinical diagnostic workflows, explicitly decouples modality semantics and physical geometry via [SEM] and [GEO]. The radar chart in (D) Right demonstrates that our zero-reference method (Concept-to-Pixel) significantly outperforms SOTA models (Spider and SR-ICL), exhibiting impressive in-domain performance and zero-shot generalization even on completely unseen modalities (, X-Ray).
• Figure 2: Overview of the proposed Concept-to-Pixel (C2P) framework. (1) The model first extracts shallow (E2) and deep (E5) backbone features to infer modality embeddings via the Style-Content Fusion Module (SCFM), which are dynamically injected into the [SEM]. (2) The universal [GEO] and modality-aware [SEM] interact with visual tokens through bidirectional Cross Attention. (3) The aggregated concepts are then fed into the Token-Guided Dynamic Head (TGDH) to generate instance-specific parameters for the final Dynamic Head to predict precise masks. To guarantee explicit decoupling, the tokens are deeply supervised by physical properties ($\mathcal{L}_{Geo}$) and MLLM-distilled knowledge ($\mathcal{L}_{Sem}$).
• Figure 3: Workflow of Geometry-Aware Inference Consensus.[GEO] compares regressed geometry with mask-derived geometry to weight each view and aggregate a robust final prediction.
• Figure 4: Qualitative comparison on representative challenging cases. From left to right: Colon Polyp, ISIC2018, TNUI, Breast, BTD, COVID, EBHI, and AMDSD. The red contours indicate the Ground Truth masks, and the yellow contours denote the model predictions.
• Figure 5: In-domain representation analysis.(Left) t-SNE of encoder outputs (E5 features). Without C2P, features from different datasets show entanglements. Our C2P effectively separates the features. (Right) t-SNE of [GEO] and [SEM]. Some geometry tokens are clustered, since samples from different dataset may have similar geometry attributes. In contrast, semantic tokens are clearly separated, indicating that each token has successfully encoded discriminative semantic concepts tied to its corresponding domain.