Continual Learning for Text Classification with Information Disentanglement Based Regularization

TL;DR

This work tackles catastrophic forgetting in continual text classification by disentangling hidden text representations into a task-generic space and a task-specific space. It learns the generic space through next sentence prediction and the specific space through task-id prediction, with separate regularization to preserve and adapt representations across tasks. A memory-efficient replay mechanism (K-means selected exemplars) complements the disentanglement, yielding improved performance over state-of-the-art baselines across multiple task sequences. The approach demonstrates robustness to varying sequence lengths and orders, and the visualizations corroborate the separation of generic and specific information. Overall, IDBR offers a principled way to balance retention and adaptation in continual NLP settings with practical memory constraints.

Abstract

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

Figures (2)

• Figure 1: Our proposed model architecture. We disentangle the hidden representation into a task generic space and a task specific space via different induction biases. When training on new tasks, different spaces are regularized separately. Also, a small portion of previous data is stored and replayed.
• Figure 2: t-SNE visualization of task generic hidden space and task specific hidden space of IDBR.