Componential Prompt-Knowledge Alignment for Domain Incremental Learning

TL;DR

This work tackles Domain Incremental Learning by identifying component-wise misalignment in domain-specific prompts as a key bottleneck for cross-domain knowledge fusion. It introduces KA-Prompt, a two-phase framework with Reusable Knowledge Mining to initialize new prompts from a memory of relevant old prompts and Aligning-Guided New Prompt Learning to preserve component alignment during training. The method demonstrates clear advantages over prior prompt-based DIL approaches across multiple benchmarks, with ablations showing the additive value of memory mining, online alignment, and adaptive fusion. The results suggest KA-Prompt improves both learning efficiency and inference robustness in multi-domain continual vision settings, offering practical benefits for cross-domain continual learning pipelines.

Abstract

Domain Incremental Learning (DIL) aims to learn from non-stationary data streams across domains while retaining and utilizing past knowledge. Although prompt-based methods effectively store multi-domain knowledge in prompt parameters and obtain advanced performance through cross-domain prompt fusion, we reveal an intrinsic limitation: component-wise misalignment between domain-specific prompts leads to conflicting knowledge integration and degraded predictions. This arises from the random positioning of knowledge components within prompts, where irrelevant component fusion introduces interference.To address this, we propose Componential Prompt-Knowledge Alignment (KA-Prompt), a novel prompt-based DIL method that introduces component-aware prompt-knowledge alignment during training, significantly improving both the learning and inference capacity of the model. KA-Prompt operates in two phases: (1) Initial Componential Structure Configuring, where a set of old prompts containing knowledge relevant to the new domain are mined via greedy search, which is then exploited to initialize new prompts to achieve reusable knowledge transfer and establish intrinsic alignment between new and old prompts. (2) Online Alignment Preservation, which dynamically identifies the target old prompts and applies adaptive componential consistency constraints as new prompts evolve. Extensive experiments on DIL benchmarks demonstrate the effectiveness of our KA-Prompt. Our source code is available at https://github.com/zhoujiahuan1991/ICML2025-KA-Prompt

Figures (9)

• Figure 1: (a) DIL aims to learn with a stream of data from different domains. State-of-the-art method C-Prompt learns domain-specific prompts independently, leading to component-aware (e.g., object part orders) misalignment. To settle this, our KA-Prompt introduces cross-domain alignment constraints. (b) During inference, by fusing prompts from different domains, our KA-Prompt outperforms C-Prompt, benefiting from improved prompt learning capacity and enhanced cross-domain knowledge compatibility.
• Figure 2: (a) Within a prompt, different components typically encode distinct types of knowledge. In C-Prompt, independently learned prompts exhibit misalignment in componential knowledge, leading to the fusion of irrelevant knowledge during inference. (b) By shuffling the componential positions of different domains before fusion, some orders with better knowledge alignment can be generated.
• Figure 3: The illustration of our KA-Prompt method. When the new domain data $D_t$ is given, the Reusable Knowledge Mining mechanism constructs a reusable prompt memory that contains the shared knowledge between the old and new domains. The reusable prompt memory is then utilized to initiate new prompts. Next, an Aligning-guided New Prompt Learning scheme is conducted for $N_{iter}$ iterations to learn the knowledge of the new domain, where the new prompt training and historical prompt online aligning ensure new knowledge acquisition and cross-domain aligning, respectively.
• Figure 4: The seen domain performance tendency along domain incremental learning process.
• Figure 5: Ablation on the model components.