Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Residual SODAP: Residual Self-Organizing Domain-Adaptive Prompting with Structural Knowledge Preservation for Continual Learning

Gyutae Oh, Jungwoo Bae, Jitae Shin

Abstract

Continual learning (CL) suffers from catastrophic forgetting, which is exacerbated in domain-incremental learning (DIL) where task identifiers are unavailable and storing past data is infeasible. While prompt-based CL (PCL) adapts representations with a frozen backbone, we observe that prompt-only improvements are often insufficient due to suboptimal prompt selection and classifier-level instability under domain shifts. We propose Residual SODAP, which jointly performs prompt-based representation adaptation and classifier-level knowledge preservation. Our framework combines $α$-entmax sparse prompt selection with residual aggregation, data-free distillation with pseudo-feature replay, prompt-usage--based drift detection, and uncertainty-aware multi-loss balancing. Across three DIL benchmarks without task IDs or extra data storage, Residual SODAP achieves state-of-the-art AvgACC/AvgF of 0.850/0.047 (DR), 0.760/0.031 (Skin Cancer), and 0.995/0.003 (CORe50).

Residual SODAP: Residual Self-Organizing Domain-Adaptive Prompting with Structural Knowledge Preservation for Continual Learning

Abstract

Continual learning (CL) suffers from catastrophic forgetting, which is exacerbated in domain-incremental learning (DIL) where task identifiers are unavailable and storing past data is infeasible. While prompt-based CL (PCL) adapts representations with a frozen backbone, we observe that prompt-only improvements are often insufficient due to suboptimal prompt selection and classifier-level instability under domain shifts. We propose Residual SODAP, which jointly performs prompt-based representation adaptation and classifier-level knowledge preservation. Our framework combines -entmax sparse prompt selection with residual aggregation, data-free distillation with pseudo-feature replay, prompt-usage--based drift detection, and uncertainty-aware multi-loss balancing. Across three DIL benchmarks without task IDs or extra data storage, Residual SODAP achieves state-of-the-art AvgACC/AvgF of 0.850/0.047 (DR), 0.760/0.031 (Skin Cancer), and 0.995/0.003 (CORe50).
Paper Structure (24 sections, 21 equations, 10 figures, 7 tables)

This paper contains 24 sections, 21 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. $\alpha$-Entmax-based residual prompt selection
  5. 3.1.1 Input and query enhancement
  6. 3.1.2 Bottleneck compression and sparse selection via $\alpha$-Entmax
  7. 3.1.3 Residual combination and output injection
  8. 3.1.4 Auxiliary Loss
  9. Statistical Knowledge Preservation via Statistics-based Pseudo Replay
  10. 3.2.1 Saving Knowledge Assets at Stage Transitions
  11. 3.2.2 Knowledge Distillation During the Next Stage Training
  12. Prompt Usage-based Drift Detection (PUDD)
  13. 3.3.1 Extracting prompt usage signals
  14. 3.3.2 Computing and measuring the drift score
  15. 3.3.3 Drift-proportional pool expansion
  16. ...and 9 more sections

Figures (10)

  • Figure 1: PCL exhibits classifier-level forgetting. We apply the cross-composition (backbone $\times$ classifier) diagnostic of liu2020more to a kim2024one-based baseline under the skin-cancer DIL setting. Residual SODAP mitigates performance degradation caused by classifier instability and improves overall accuracy. The experimental setup and interpretation are provided in Sup. \ref{['supp:A']}--\ref{['supp:B']}.
  • Figure 2: Overview of Residual SODAP.
  • Figure 3: Stage-wise prompt selection with PUDD-triggered pool expansion (60$\rightarrow$84$\rightarrow$94) and sparse $\alpha$-entmax routing on the DR dataset. Corresponding results on the other datasets are provided in Sup. \ref{['supp:E']}.
  • Figure 4: Prompt usage and optimization dynamics on the DR dataset: (a) balanced selection, (b) no redundancy after expansion, (c--d) uncertainty-weighted loss rebalancing across stages. Corresponding results for the other datasets are provided in Sup. \ref{['supp:E']}.
  • Figure 5: Results of cross-stage backbone--classifier compositions ($3\times 3$) under the skin-cancer DIL protocol. The baseline visualization follows the prompt-based continual learning method of kim2024one.
  • ...and 5 more figures