Information-Theoretic Complementary Prompts for Improved Continual Text Classification

TL;DR

This work tackles continual text classification (CTC) by addressing catastrophic forgetting and forward knowledge transfer. It introduces Information-Theoretic Complementary Prompts (InfoComp), which deploy two prompts: a task-specific P-Prompt and a task-invariant S-Prompt, guided by an information-theoretic mutual-information framework to learn more informative prompts without data replay. Two MI-based losses are proposed: $\mathcal{L}^{\text{p-info}}$ to strengthen task-specific knowledge in the P-Prompt and $\mathcal{L}^{\text{s-info}}$ to preserve shared knowledge in the S-Prompt, enabling robust CF mitigation and improved FKT. Extensive experiments on standard and long-sequence CTC benchmarks show that InfoComp consistently outperforms prior state-of-the-art methods, demonstrating effective continual learning with a compact, scalable prompting scheme and strong forward transfer capabilities.

Abstract

Continual Text Classification (CTC) aims to continuously classify new text data over time while minimizing catastrophic forgetting of previously acquired knowledge. However, existing methods often focus on task-specific knowledge, overlooking the importance of shared, task-agnostic knowledge. Inspired by the complementary learning systems theory, which posits that humans learn continually through the interaction of two systems -- the hippocampus, responsible for forming distinct representations of specific experiences, and the neocortex, which extracts more general and transferable representations from past experiences -- we introduce Information-Theoretic Complementary Prompts (InfoComp), a novel approach for CTC. InfoComp explicitly learns two distinct prompt spaces: P(rivate)-Prompt and S(hared)-Prompt. These respectively encode task-specific and task-invariant knowledge, enabling models to sequentially learn classification tasks without relying on data replay. To promote more informative prompt learning, InfoComp uses an information-theoretic framework that maximizes mutual information between different parameters (or encoded representations). Within this framework, we design two novel loss functions: (1) to strengthen the accumulation of task-specific knowledge in P-Prompt, effectively mitigating catastrophic forgetting, and (2) to enhance the retention of task-invariant knowledge in S-Prompt, improving forward knowledge transfer. Extensive experiments on diverse CTC benchmarks show that our approach outperforms previous state-of-the-art methods.

Figures (4)

• Figure 1: InfoComp learns two distinct prompt types: P-Prompt and S-Prompt. The P-Prompt encodes task-specific knowledge to reduce CF, while the S-Prompt captures task-invariant knowledge to enhance FKT. "Trm" represents the Transformer encoder block within the PLM.
• Figure 2: Alleviating the issue of forgetting shared knowledge across tasks by maximizing the MI $I(V^k_i, V^{k'}_i)$.
• Figure 3: (a) SimSiam for the positive sample pair. (b) SimSiam for the prompt-augmented positive sample pair.
• Figure 4: Comparison of task-wise accuracy for the task sequences Order3 and Order4. Our InfoComp consistently outperforms ProgPrompt razdaibiedina2022progressive, across all task-wise evaluations.