Leakage and Interpretability in Concept-Based Models

TL;DR

This work formalizes leakage in concept-based models using an information-theoretic lens, introducing CTL and ICL to quantify how task information and interconcept information leak into learned concepts. It demonstrates that CTL and ICL are strong predictors of model behavior under interventions and robust against prior interpretability metrics, revealing substantial leakage in Concept Embedding Models (CEMs) regardless of hyperparameters. The study identifies key leakage causes—insufficient supervision, over-expressive encodings, incomplete concept sets, and misspecified heads—and provides practical guidelines to mitigate leakage and preserve interpretability. By releasing leakage-score tooling and validating across synthetic datasets, the work advocates routine leakage assessment as a core step in designing interpretable concept-based systems for high-stakes applications.

Abstract

Concept Bottleneck Models aim to improve interpretability by predicting high-level intermediate concepts, representing a promising approach for deployment in high-risk scenarios. However, they are known to suffer from information leakage, whereby models exploit unintended information encoded within the learned concepts. We introduce an information-theoretic framework to rigorously characterise and quantify leakage, and define two complementary measures: the concepts-task leakage (CTL) and interconcept leakage (ICL) scores. We show that these measures are strongly predictive of model behaviour under interventions and outperform existing alternatives in robustness and reliability. Using this framework, we identify the primary causes of leakage and provide strong evidence that Concept Embedding Models exhibit substantial leakage regardless of the hyperparameters choice. Finally, we propose practical guidelines for designing concept-based models to reduce leakage and ensure interpretability.

Figures (24)

• Figure 1: Scheme of CBM (left) and CEM (right) architectures. Quantities with green and blue backgrounds are predicted scalars and vectors respectively.
• Figure 2: Distributions of predicted activations for concept 1 on the test set in CBMs with increasing leakage from left to right. Colours are based on the ground-truth values of concepts and task label, and to improve visibility, bars with lower counts are rendered in front of those with higher counts.
• Figure 3: Performance upon random intervention of the three models analysed in Figure \ref{['figure_TT_leakage_concept_distributions']}. See Appendix \ref{['App_experiments_Scores_evaluation']} for more details.
• Figure 4: Intervention performance and leakage scores computed for pairs of soft and logit CBMs with different amounts of leakage but similar evaluation scores (see Table \ref{['table_evaluation_scores_single_models']}) trained on TabularToy(0.25), dSprites(0) and 3dshapes(0). Metrics are evaluated on a 5-fold basis for each model.
• Figure 5: Leakage scores evaluated on hard CBMs on datasets with high and low ground-truth interconcept MI. The 95% confidence intervals are obtained from 5-fold training on each dataset.