SPEGC: Continual Test-Time Adaptation via Semantic-Prompt-Enhanced Graph Clustering for Medical Image Segmentation

Abstract

In medical image segmentation tasks, the domain gap caused by the difference in data collection between training and testing data seriously hinders the deployment of pre-trained models in clinical practice. Continual Test-Time Adaptation (CTTA) aims to enable pre-trained models to adapt to continuously changing unlabeled domains, providing an effective approach to solving this problem. However, existing CTTA methods often rely on unreliable supervisory signals, igniting a self-reinforcing cycle of error accumulation that culminates in catastrophic performance degradation. To overcome these challenges, we propose a CTTA via Semantic-Prompt-Enhanced Graph Clustering (SPEGC) for medical image segmentation. First, we design a semantic prompt feature enhancement mechanism that utilizes decoupled commonality and heterogeneity prompt pools to inject global contextual information into local features, alleviating their susceptibility to noise interference under domain shift. Second, based on these enhanced features, we design a differentiable graph clustering solver. This solver reframes global edge sparsification as an optimal transport problem, allowing it to distill a raw similarity matrix into a refined and high-order structural representation in an end-to-end manner. Finally, this robust structural representation is used to guide model adaptation, ensuring predictions are consistent at a cluster-level and dynamically adjusting decision boundaries. Extensive experiments demonstrate that SPEGC outperforms other state-of-the-art CTTA methods on two medical image segmentation benchmarks. The source code is available at https://github.com/Jwei-Z/SPEGC-for-MIS.

Figures (7)

• Figure 1: Conceptual comparison of different paradigms in CTTA. Compared to (A) Prompt adaptation only and (B) Source Prototype Alignment, (C) our SPEGC innovatively leverages graph clustering to extract structural information, achieving "Good" performance across Performance (Perf), Generalization (Gen), Error Accumulation (EA), and Catastrophic Forgetting (CF), while maintaining an acceptable Computational Complexity (CC).
• Figure 2: Overview of the SPEGC. For the continual target domains stream, we first extract local node features ($V$) and construct a Pseudo mini-batch via a feature queue (Enqueue/Dequeue). Subsequently, the Semantic Prompt Feature Enhancement (SPFE, \ref{['sec:SPFE']}) module utilizes Attention Pooling to generate a Query, retrieving information from the decoupled commonality ($P_{CO}$) and heterogeneity ($P_{HE}$) prompt pools to generate the enhanced batch features $V^{\star}$, thereby mitigating noisy information from the domain shift. Based on these features, the Differentiable Graph Clustering Solver (DGCS, \ref{['sec:DGCS']}) computes the initial edge similarity matrix $S^{\prime}$ and refines it end-to-end into $S^{\star}$. Finally, the segmentation network is fine-tuned via backpropagation to achieve efficient adaptation (\ref{['sec:LF']}).
• Figure 3: t-SNE maaten2008visualizing visualization of the embeddings learned by D-Prompt and SEC-Prompt on the OD/OC task. Different colors represent images from different domains.
• Figure 4: Performance of SPEGC with various Z on the OD/OC segmentation task.
• Figure 5: Performance of our SPEGC with various M on the OD/OC segmentation task.

Theorems & Definitions (3)

• Definition 1: Discrete Global Consistency
• Lemma 1: Nature of the Unregularized Solution
• Theorem 1: Consequence of Entropy Regularization