Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features?

TL;DR

This work investigates whether Concept Bottleneck Models can learn semantically meaningful input features by aligning concept predictions with human-interpretable input representations. It demonstrates that semantic mappings emerge when concept annotations are accurate and concept co-occurrence is varied, using synthetic playing cards and CheXpert datasets to contrast instance-level and class-level concept configurations. Saliency analyses and the Oracle Impurity Score reveal that instance-level concept annotations facilitate meaningful feature attribution and reduce unintended inter-concept correlations, whereas class-level configurations can hinder semantic mapping. The findings advocate careful dataset design and instance-level concept annotations to realize the interpretability promises of CBMs and suggest future exploration with scalable, semantically grounded annotations or language–vision methods.

Abstract

Concept Bottleneck Models (CBMs) are regarded as inherently interpretable because they first predict a set of human-defined concepts which are used to predict a task label. For inherent interpretability to be fully realised, and ensure trust in a model's output, it's desirable for concept predictions to use semantically meaningful input features. For instance, in an image, pixels representing a broken bone should contribute to predicting a fracture. However, current literature suggests that concept predictions often rely on irrelevant input features. We hypothesise that this occurs when dataset labels include inaccurate concept annotations, or the relationship between input features and concepts is unclear. In general, the effect of dataset labelling on concept representations remains an understudied area. In this paper, we demonstrate that CBMs can learn to map concepts to semantically meaningful input features, by utilising datasets with a clear link between the input features and the desired concept predictions. This is achieved, for instance, by ensuring multiple concepts do not always co-occur and, therefore provide a clear training signal for the CBM to distinguish the relevant input features for each concept. We validate our hypothesis on both synthetic and real-world image datasets, and demonstrate under the correct conditions, CBMs can learn to attribute semantically meaningful input features to the correct concept predictions.

Figures (30)

• Figure 1: Samples from the playing cards dataset
• Figure 2: Example CheXpert sample with concept annotations
• Figure 3: Positive relevance (red) is always applied to the expected input features (playing cards) when concept supervision is used in training.
• Figure 4: Positive saliency (red) does not always map to the semantically meaningful input features for each concept for class-level poker cards.
• Figure 5: Random cards and poker cards both have a high positive proportion of relevance, indicating the models have learned to use semantically meaningful input features to predict concepts, unlike standard NN and most concepts for class-level poker cards.