Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation

TL;DR

A class-aware domain Knowledge Fusion and Fission method for continual test-time adaptation is proposed, called KFF, which adaptively expands and merges class-aware domain knowledge in old and new domains according to the test-time data from different domains, where discriminative historical knowledge can be dynamically accumulated.

Abstract

Continual Test-Time Adaptation (CTTA) aims to quickly fine-tune the model during the test phase so that it can adapt to multiple unknown downstream domain distributions without pre-acquiring downstream domain data. To this end, existing advanced CTTA methods mainly reduce the catastrophic forgetting of historical knowledge caused by irregular switching of downstream domain data by restoring the initial model or reusing historical models. However, these methods are usually accompanied by serious insufficient learning of new knowledge and interference from potentially harmful historical knowledge, resulting in severe performance degradation. To this end, we propose a class-aware domain Knowledge Fusion and Fission method for continual test-time adaptation, called KFF, which adaptively expands and merges class-aware domain knowledge in old and new domains according to the test-time data from different domains, where discriminative historical knowledge can be dynamically accumulated. Specifically, considering the huge domain gap within streaming data, a domain Knowledge FIssion (KFI) module is designed to adaptively separate new domain knowledge from a paired class-aware domain prompt pool, alleviating the impact of negative knowledge brought by old domains that are distinct from the current domain. Besides, to avoid the cumulative computation and storage overheads from continuously fissioning new knowledge, a domain Knowledge FUsion (KFU) module is further designed to merge the fissioned new knowledge into the existing knowledge pool with minimal cost, where a greedy knowledge dynamic merging strategy is designed to improve the compatibility of new and old knowledge while keeping the computational efficiency. Extensive experiments on the ImageNet-C dataset verify the effectiveness of our proposed method against other methods.

Figures (11)

• Figure 1: Existing methods DBLP:journals/corr/abs-2403-10650 failed to accumulate knowledge in distinct domains, where conflicting knowledge disrupts the gradient direction during optimizing and remains a severe domain conflict problem.
• Figure 2: The Overall pipeline of our KFF with Knowledge Fission(KFI) and Knowledge Fusion(KFU). For each test batch, the KFI module dynamically fissions a domain prompt and several class prompts to adjust the source model. Sequentially, KFU merges the fissioned prompts after optimizing, achieving the balance between effectiveness and efficiency.
• Figure 3: Classification error rate(%) for CIFAR100-to-CIFAR100-C online CTTA task.
• Figure 4: Computational analysis on ImageNet-C.
• Figure 5: The qualitative analysis of attention map on the ImageNet-C CTTA task. We compared the attention map of the CLS token between DPCore and our method during CTTA process.

Theorems & Definitions (7)

• Definition A.1: Well-Separated Clusters
• Lemma A.1
• proof
• Lemma A.2
• proof
• Proposition A.3
• proof