Towards Robust and Reliable Concept Representations: Reliability-Enhanced Concept Embedding Model

TL;DR

This work tackles reliability gaps in concept bottleneck models by addressing two core weaknesses: sensitivity to concept-irrelevant background features and semantic inconsistency of the same concept across samples. It introduces RECEM, a framework combining Concept-Level Disentanglement with a Gradient Reversal Layer and HSIC regularization, and Concept Mixup for semantic alignment across samples, optimized via a joint loss that includes reconstruction and alignment terms. Empirical results across CUB, TravelingBirds, CelebA, and AwA2 show that RECEM surpasses strong baselines in concept accuracy, task accuracy, and Concept Alignment Score (CAS), particularly under background shifts and incomplete annotations, while enabling more faithful human interventions. The findings underscore the value of disentangling nuisance information and aligning concept semantics to improve both interpretability and robustness in CBMs for real-world deployment.

Abstract

Concept Bottleneck Models (CBMs) aim to enhance interpretability by predicting human-understandable concepts as intermediates for decision-making. However, these models often face challenges in ensuring reliable concept representations, which can propagate to downstream tasks and undermine robustness, especially under distribution shifts. Two inherent issues contribute to concept unreliability: sensitivity to concept-irrelevant features (e.g., background variations) and lack of semantic consistency for the same concept across different samples. To address these limitations, we propose the Reliability-Enhanced Concept Embedding Model (RECEM), which introduces a two-fold strategy: Concept-Level Disentanglement to separate irrelevant features from concept-relevant information and a Concept Mixup mechanism to ensure semantic alignment across samples. These mechanisms work together to improve concept reliability, enabling the model to focus on meaningful object attributes and generate faithful concept representations. Experimental results demonstrate that RECEM consistently outperforms existing baselines across multiple datasets, showing superior performance under background and domain shifts. These findings highlight the effectiveness of disentanglement and alignment strategies in enhancing both reliability and robustness in CBMs.

Figures (8)

• Figure 1: (a) The model predicts $\hat{c}$ and $\hat{c}'$, corresponding to the original image and the image with a background shift. Reliable concept representations should be robust to irrelevant variations and exhibit consistent semantic meanings. (b) shows the cosine similarity distribution between embeddings before and after the background change, reflecting model sensitivity to irrelevant features. The results indicate that CEM (blue) has lower similarity, reflecting a lack of consistency and sensitivity to irrelevant features. (c) presents the similarity distribution of the same concept representation across different samples in the CUB dataset, illustrating semantic consistency. It demonstrates that CEM (blue) exhibits poor semantic consistency across samples.
• Figure 2: An intuitive illustration of the proposed mechanisms, Concept-Level Disentanglement and Concept Mixup.
• Figure 3: Overview of the proposed RECEM architecture, highlighting three key components with numbered dashed boxes: (1)Concept-Level Disentanglement, where $E_{\text{dis}}$ extracts concept-irrelevant features $\hat{z}_i$ and $D_{\text{rec}}$ ensures rich semantic information within the concept embeddings; (2)Concept Mixup, a mechanism aligning and proportionally mixing true concept embeddings (indicated by the green dashed box) across samples to achieve semantic consistency of concepts; and (3)Illustration of Concept Mixup, where arrows demonstrate the Concept Mixup mechanism enhances the consistency of semantic representations for the same concept after alignment.
• Figure 4: The impact of the annealing coefficient $\beta$
• Figure 5: Task accuracy under varying levels of human intervention in concept predictions for different models.