TP-Seg: Task-Prototype Framework for Unified Medical Lesion Segmentation

Abstract

Building a unified model with a single set of parameters to efficiently handle diverse types of medical lesion segmentation has become a crucial objective for AI-assisted diagnosis. Existing unified segmentation approaches typically rely on shared encoders across heterogeneous tasks and modalities, which often leads to feature entanglement, gradient interference, and suboptimal lesion discrimination. In this work, we propose TP-Seg, a task-prototype framework for unified medical lesion segmentation. On one hand, the task-conditioned adapter effectively balances shared and task-specific representations through a dual-path expert structure, enabling adaptive feature extraction across diverse medical imaging modalities and lesion types. On the other hand, the prototype-guided task decoder introduces learnable task prototypes as semantic anchors and employs a cross-attention mechanism to achieve fine-grained modeling of task-specific foreground and background semantics. Without bells and whistles, TP-Seg consistently outperforms specialized, general and unified segmentation methods across 8 different medical lesion segmentation tasks covering multiple imaging modalities, demonstrating strong generalization, scalability and clinical applicability.

Figures (8)

• Figure 1: Comparison of framework design and average performance between previous unified models SR-ICLseggptspidersam2unet and our TP-Seg. (1) Single-path shared encoder vs. Dual-path routing encoder; (2) Task-agnostic decoder vs. Prototype-guided foreground-background decoder; (3) Average Dice (%): 80.75 vs. Average Dice (%): 86.63; (4) Average mIoU (%): 82.83 vs. Average mIoU (%): 86.44.
• Figure 2: Overall architecture of the proposed TP-Seg framework for unified medical lesion segmentation. Each input image, together with its task embedding, is processed by the task-conditioned routing block (TCRB) for feature extraction, followed by the prototype-guided task decoder (PGTD) for task-aware decoding and final lesion prediction.
• Figure 3: Illustration of the task-conditioned routing block (TCRB), which integrates a shared router and a task-specific router within each task-conditioned adapter (TCA) to achieve dynamic feature adaptation before the frozen encoder blocks.
• Figure 4: Illustration of the prototype-guided task decoder.
• Figure 5: Visual comparison of TP-Seg with other unified models, including Spider spider, SAM2-UNet sam2-UNet ,SegGPT seggpt and SR-ICL SR-ICL, across the 8 medical lesion segmentation tasks.