Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions

TL;DR

The paper tackles interpretability and robustness in AI by marrying causal representation learning (CRL) with a principled, label-efficient alignment to human concepts. It introduces two alignment estimators—a linear Group Lasso and a kernelized, non-parametric variant—each providing formal guarantees on concept correctness and label efficiency without requiring interventions. Theoretical results establish finite-sample bounds and asymptotic consistency for the permutation alignment, while experiments on synthetic and image datasets show reduced concept impurity and competitive downstream accuracy with far fewer concept labels than standard CBMs. The work advances practical, theory-grounded concept learning, enabling more reliable and interpretable AI systems in settings with correlated concepts.

Abstract

Machine learning is a vital part of many real-world systems, but several concerns remain about the lack of interpretability, explainability and robustness of black-box AI systems. Concept Bottleneck Models (CBM) address some of these challenges by learning interpretable concepts from high-dimensional data, e.g. images, which are used to predict labels. An important issue in CBMs are spurious correlation between concepts, which effectively lead to learning "wrong" concepts. Current mitigating strategies have strong assumptions, e.g., they assume that the concepts are statistically independent of each other, or require substantial interaction in terms of both interventions and labels provided by annotators. In this paper, we describe a framework that provides theoretical guarantees on the correctness of the learned concepts and on the number of required labels, without requiring any interventions. Our framework leverages causal representation learning (CRL) methods to learn latent causal variables from high-dimensional observations in a unsupervised way, and then learns to align these variables with interpretable concepts with few concept labels. We propose a linear and a non-parametric estimator for this mapping, providing a finite-sample high probability result in the linear case and an asymptotic consistency result for the non-parametric estimator. We evaluate our framework in synthetic and image benchmarks, showing that the learned concepts have less impurities and are often more accurate than other CBMs, even in settings with strong correlations between concepts.

Figures (23)

• Figure 1: Left: An overview of our framework: we learn the alignment function $\alpha$ that maps causal representations $M_i$ learned on cheap unlabelled data by a causal representation learning (CRL) encoder, to interpretable concepts $C_j$ using only few concept labels. As in standard CBMs, these concepts are used in a downstream task like regression or classification of $Y$. Right: Data generating process, where $G$ are the latent causal variables, $X$ is an observation, $M$ are the representations learned by a model $g_\psi$, $C$ are the interpretable concepts and $Y$ is the final label.
• Figure 2: Permutation error rate for spline features. From top left to bottom right we vary: (i) the regularization parameter, (ii) the number of dimensions, (iii) the correlation of the variables and (iv) the number of labels. The first plot of each pair shows the wellspecified and the second the misspecified case. We average over $10$ seeds and shade the 25-75th percentile.
• Figure 3: Execution times for our estimators and several baseline models on the Temporal Causal3DIdent dataset. Left: The causal variables are continuous-valued. The baseline is given by training several neural networks and the matching is based on the $R^2$-scores. We report the time needed to estimate the matching and train the neural networks. Right: The causal variables are binarized and a downstream classification task is added. We report the time needed to estimate the matching and learn the classification task or the time needed to train the concept-based models.
• Figure 4: The different types of diffeomorphisms used in the misspecified case.
• Figure 5: Examples of the $7$ shapes in the Temporal Causal3DIdent dataset. From left to right: teapot, armadillow, bunny, cow, dragon, head and horse.

Theorems & Definitions (16)

• Definition 3.1
• Theorem 4.2
• Corollary 4.2
• Theorem 5.1
• Theorem 5.2
• Lemma 5.2
• Theorem A.2
• Lemma A.4
• Theorem A.5
• Theorem A.5