Language Guided Concept Bottleneck Models for Interpretable Continual Learning

TL;DR

This work tackles the dual challenge of mitigating catastrophic forgetting and maintaining interpretability in continual learning. It introduces Language-Guided Concept Bottleneck Models (LG-CBM) that leverage CLIP-aligned language concepts to form a transparent bottleneck, while a semantic-guided prototype augmentation mechanism mitigates forgetting by generating plausible pseudo-features for old classes. The model aligns the Concept Bottleneck Layer with CLIP activations through a similarity loss and enforces sparsity in the final classifier to promote interpretable decision-making. Extensive experiments across coarse- and fine-grained datasets show competitive or superior performance compared to state-of-the-art methods, accompanied by concept visualizations that reveal the reasoning behind predictions. This approach enables interpretable continual learning with strong robustness to forgetting, achieved through language-informed concept representations and task-aware prototype augmentation.

Abstract

Continual learning (CL) aims to enable learning systems to acquire new knowledge constantly without forgetting previously learned information. CL faces the challenge of mitigating catastrophic forgetting while maintaining interpretability across tasks. Most existing CL methods focus primarily on preserving learned knowledge to improve model performance. However, as new information is introduced, the interpretability of the learning process becomes crucial for understanding the evolving decision-making process, yet it is rarely explored. In this paper, we introduce a novel framework that integrates language-guided Concept Bottleneck Models (CBMs) to address both challenges. Our approach leverages the Concept Bottleneck Layer, aligning semantic consistency with CLIP models to learn human-understandable concepts that can generalize across tasks. By focusing on interpretable concepts, our method not only enhances the models ability to retain knowledge over time but also provides transparent decision-making insights. We demonstrate the effectiveness of our approach by achieving superior performance on several datasets, outperforming state-of-the-art methods with an improvement of up to 3.06% in final average accuracy on ImageNet-subset. Additionally, we offer concept visualizations for model predictions, further advancing the understanding of interpretable continual learning.

Figures (8)

• Figure 1: The framework of our method for task t. The concept alignment module aligns the concept score matrix with the CLIP concept activation matrix. The semantic-guided prototype augmentation module leverages semantic knowledge to identify the most semantically similar class within $\mathcal{Y}_t$, generating pseudo-features for previously learned classes to mitigate catastrophic forgetting.
• Figure 2: Performance-Parameter comparison of our method and benchmark methods on ImageNet-subset B-10 Inc-10.
• Figure 3: (a) Contribution visualization of axolotl after training on ImageNet-subset B-10 Inc-10. (b) Contribution visualization of sayornis after training on CUB-200 B-10 B-10.
• Figure 4: Top-7 concepts change of gazania after training the first and all the tasks on Flower B-25 Inc-25.
• Figure 5: Average incremental accuracy $\bar{A}$ curve of our method with other state-of-the-art methods across all benchmark datasets.