Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learning

TL;DR

This work tackles information leakage in Concept Bottleneck Models by introducing SupCBM, a paradigm where concept predictions are explicitly supervised by class labels and integrated via a fixed intervention matrix. SupCBM employs a two-level hierarchical concept set of perceptual nouns and descriptive adjectives, constructed with GPT prompts, and trained with label-aware supervision and a concept pooling mechanism to minimize leakage while preserving predictive power. The label for a class is computed as the sum of probabilities of involved concepts, $l_j = \sum_{i=1}^{pq} c_i \cdot \mathcal{I}_{i,j}$, with a training objective $\min \left( \alpha \cdot \ell_{BCE}(c, GT_c) + (1-\alpha) \cdot \ell_{CE}(c \mathcal{I}, GT_l) \right)$, removing the need for a separate label predictor. Empirically, SupCBM outperforms state-of-the-art CBMs across CIFAR-10/100, CUB-Bird, and HAM10000, approaches the performance of feature-based baselines, and shows higher resilience to leakage and stronger generality across backbones, while providing interpretable, intervention-friendly predictions.

Abstract

Concept Bottleneck Models (CBMs) aim to deliver interpretable and interventionable predictions by bridging features and labels with human-understandable concepts. While recent CBMs show promising potential, they suffer from information leakage, where unintended information beyond the concepts (either when concepts are represented with probabilities or binary states) are leaked to the subsequent label prediction. Consequently, distinct classes are falsely classified via indistinguishable concepts, undermining the interpretation and intervention of CBMs. This paper alleviates the information leakage issue by introducing label supervision in concept predication and constructing a hierarchical concept set. Accordingly, we propose a new paradigm of CBMs, namely SupCBM, which achieves label predication via predicted concepts and a deliberately-designed intervention matrix. SupCBM focuses on concepts that are mostly relevant to the predicted label and only distinguishes classes when different concepts are presented. Our evaluations show that SupCBM outperforms SOTA CBMs over diverse datasets. It also manifests better generality across different backbone models. With proper quantification of information leakage in different CBMs, we demonstrate that SupCBM significantly reduces the information leakage.

Figures (4)

• Figure 1: Workflow of SupCBM. The training of SupCBM consists three mains stages.
• Figure 2: Information leakage evaluation. If a CBM manifests higher resilience to information leakage, its performance should drop more quickly when more concepts are removed.
• Figure 3: Generality evaluation.
• Figure 4: Case study of SupCBM on CUB-Bird and FLOWER datasets.