Incremental Residual Concept Bottleneck Models

TL;DR

The paper addresses incomplete concept banks in Concept Bottleneck Models (CBMs) by proposing Incremental Residual Concept Bottleneck Model (Res-CBM), which adds learnable residual concepts $r=U f$ to the base bottleneck $c=W f$ and translates them into human-interpretable concepts via an incremental discovery process. It introduces Concept Utilization Efficiency (CUE) to quantify the trade-off between accuracy, concept count, and concept length, and presents a post-hoc framework applicable to any CBM and concept bank. Through experiments on seven datasets, Res-CBM achieves higher accuracy and efficiency than state-of-the-art CLIP-based CBMs and matches or exceeds black-box models in several settings, with notable gains in few-shot scenarios. Overall, the method improves CBM completeness and interpretability while preserving predictive performance, offering a scalable path toward more trustworthy AI systems.

Abstract

Concept Bottleneck Models (CBMs) map the black-box visual representations extracted by deep neural networks onto a set of interpretable concepts and use the concepts to make predictions, enhancing the transparency of the decision-making process. Multimodal pre-trained models can match visual representations with textual concept embeddings, allowing for obtaining the interpretable concept bottleneck without the expertise concept annotations. Recent research has focused on the concept bank establishment and the high-quality concept selection. However, it is challenging to construct a comprehensive concept bank through humans or large language models, which severely limits the performance of CBMs. In this work, we propose the Incremental Residual Concept Bottleneck Model (Res-CBM) to address the challenge of concept completeness. Specifically, the residual concept bottleneck model employs a set of optimizable vectors to complete missing concepts, then the incremental concept discovery module converts the complemented vectors with unclear meanings into potential concepts in the candidate concept bank. Our approach can be applied to any user-defined concept bank, as a post-hoc processing method to enhance the performance of any CBMs. Furthermore, to measure the descriptive efficiency of CBMs, the Concept Utilization Efficiency (CUE) metric is proposed. Experiments show that the Res-CBM outperforms the current state-of-the-art methods in terms of both accuracy and efficiency and achieves comparable performance to black-box models across multiple datasets.

Figures (6)

• Figure 1: Challenges in CBMs. When a concept contains many atomic attributes, making it too complex for human to comprehend. When a concept is too high-level, DNNs may struggle to provide accurate prediction. Additionally, there is a risk of human-designed concept bank missing some important concepts.
• Figure 2: Different CBMs' structures. (a): CLIP linear probing radford2021learning. (b): Original Concept Bottleneck Model (CBM) koh2020concept. (c) Post-hoc Concept Bottleneck Model (PCBM) yang2023languageoikarinen2023labelyuksekgonul2022post. (d): Hybrid Post-hoc Concept Bottleneck Model (PCBM-h) yuksekgonul2022post.
• Figure 3: Incremental Residual Concept Bottleneck Model.
• Figure 4: Test accuracy (%) comparison between Res-CBM and LP on 6 datasets. The x-axis represents the number of labeled images.
• Figure 5: Ablation results on residual vector numbers, concept similarity loss weights and candidate concept numbers.