Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation

TL;DR

This work tackles continual test-time adaptation (CTTA) by redesigning the online data buffering process to operate in a source-free, unsupervised, single-pass data stream. It introduces an uncertainty-aware buffer to collect high-certainty samples, a graph-based Class Relation Preservation Constraint (CRP) to maintain intrinsic semantic relations and mitigate forgetting, and a pseudo-target replay objective to curb error accumulation. The approach is evaluated on both segmentation and classification CTTA tasks, showing consistent improvements over prior methods across multiple datasets and settings, including long-term Cityscapes-to-ACDC and ImageNet-/CIFAR-based benchmarks. The results demonstrate robust continual adaptation with stronger semantic preservation and reduced forgetting, highlighting practical benefits for real-world deployment under continual domain shifts.

Abstract

Continual Test-Time Adaptation (CTTA) involves adapting a pre-trained source model to continually changing unsupervised target domains. In this paper, we systematically analyze the challenges of this task: online environment, unsupervised nature, and the risks of error accumulation and catastrophic forgetting under continual domain shifts. To address these challenges, we reshape the online data buffering and organizing mechanism for CTTA. We propose an uncertainty-aware buffering approach to identify and aggregate significant samples with high certainty from the unsupervised, single-pass data stream. Based on this, we propose a graph-based class relation preservation constraint to overcome catastrophic forgetting. Furthermore, a pseudo-target replay objective is used to mitigate error accumulation. Extensive experiments demonstrate the superiority of our method in both segmentation and classification CTTA tasks. Code is available at https://github.com/z1358/OBAO.

Figures (4)

• Figure 1: Comparison of different buffering mechanisms. (a) Without a buffer, only unreliable signals from the current unsupervised input data are available. (b) Some methodsDobler_2023_CVPRSojka_2023_ICCV use a static source buffer that stores source samples with ground-truth labels to provide supervised signals. (c) We work on a dynamic target buffer, which can provide reliable signals for the adaptation process.
• Figure 2: The framework of our proposed method. At time step $t$, the incoming data and sampled buffer data are used for adaptation. The proposed class relation preservation loss $\mathcal{L}_\textit{CRP}$ ensures topological consistency between the Class Relation Graph (CRG) $G_t$ estimated at the current time step $t$ and $\hat{G}$ from the source domain, thereby effectively preserving intrinsic semantic information. Concurrently, we integrate a pseudo-target replay loss $\mathcal{L}_\textit{PCE}$ to mitigate error accumulation.
• Figure 3: Qualitative comparison of semantic segmentation on the Cityscapes-to-ACDC CTTA task. The results for all methods come from the final round.
• Figure 5: The CAM visualizations. These images belong to the hoopskirt category.