A3-TTA: Adaptive Anchor Alignment Test-Time Adaptation for Image Segmentation

TL;DR

A3-TTA addresses the core challenge of test-time adaptation for image segmentation under domain shift without access to source data or multi-epoch training. It introduces Anchor-Target Images (ATIs) identified via Class Compact Density (CCD) and stored in a dynamic feature bank to guide refined pseudo-labels through feature alignment (FAR) and boundary-aware supervision, complemented by a self-adaptive mean-teacher (SER) strategy for stable single-pass updates. The framework demonstrates substantial improvements over state-of-the-art TTA methods on cardiac MRI, prostate MRI, and adverse-conditions Cityscapes, with strong continual-learning performance and anti-forgetting capabilities. These results illustrate the practical impact of ATI-guided adaptation for robust, real-time segmentation across diverse domains, while revealing avenues for extending to 3D segmentation and cross-modality shifts.

Abstract

Test-Time Adaptation (TTA) offers a practical solution for deploying image segmentation models under domain shift without accessing source data or retraining. Among existing TTA strategies, pseudo-label-based methods have shown promising performance. However, they often rely on perturbation-ensemble heuristics (e.g., dropout sampling, test-time augmentation, Gaussian noise), which lack distributional grounding and yield unstable training signals. This can trigger error accumulation and catastrophic forgetting during adaptation. To address this, we propose \textbf{A3-TTA}, a TTA framework that constructs reliable pseudo-labels through anchor-guided supervision. Specifically, we identify well-predicted target domain images using a class compact density metric, under the assumption that confident predictions imply distributional proximity to the source domain. These anchors serve as stable references to guide pseudo-label generation, which is further regularized via semantic consistency and boundary-aware entropy minimization. Additionally, we introduce a self-adaptive exponential moving average strategy to mitigate label noise and stabilize model update during adaptation. Evaluated on both multi-domain medical images (heart structure and prostate segmentation) and natural images, A3-TTA significantly improves average Dice scores by 10.40 to 17.68 percentage points compared to the source model, outperforming several state-of-the-art TTA methods under different segmentation model architectures. A3-TTA also excels in continual TTA, maintaining high performance across sequential target domains with strong anti-forgetting ability. The code will be made publicly available at https://github.com/HiLab-git/A3-TTA.

Figures (6)

• Figure 1: Motivation of our A3-TTA. (a) BatchNorm Adaptation suffers from feature misalignment due to mismatched statistics between source and target domains; (b) Unsupervised Optimization, such as entropy minimization, may lead to catastrophic forgetting of source knowledge; (c) Pseudo-Label Optimization is prone to unreliable pseudo labels and error accumulation; (d) Our Adaptive Anchor Alignment-based Adaptation (A3-TTA) mitigates these issues by leveraging Anchor Target Images (ATIs) to establish a feature bank, facilitating more stable feature alignment and generating reliable pseudo labels for adaptation.
• Figure 2: Overview of our A3-TTA. A dynamic feature bank is constructed using anchor-target samples selected based on CCD. For a test image $x_i$, the most similar feature from the bank is fetched to obtain refined pseudo labels that are used for supervision via the semantic loss $L_{sem}$ and the boundary-guided entropy minimization loss $L_{be}$. A self-adaptive mean teacher framework is also employed for more robust adaptation through the loss $L_{mt}$. After each iteration update, the student model generates the final predictions for the current test images.
• Figure 3: Qualitative comparison of different TTA methods for (a) cardiac image segmentation and (b) prostate MRI segmentation across multiple target domains.
• Figure 4: Sensitivity analysis of hyper-parameters on Domain B of the M$\&$M dataset.
• Figure 5: Effectiveness of Anchor-Target Images (ATIs ). (a) T-SNE visualization of features of source images and target images (ATIs and non-ATIs) obtained by the source model. (b) T-SNE visualization of feature alignment between non-ATIs and ATIs after applying our A3-TTA. (c) Performance comparison between before and after A3-TTA, and quality of ATIs filtered by different methods.