From Black-box to Causal-box: Towards Building More Interpretable Models

TL;DR

This paper defines causal interpretability as the ability to evaluate counterfactual predictions from observational data and shows that standard blackbox and many concept-based models are not causally interpretable in general. It then builds a principled framework using augmented structural causal models (ASCM) and a generalized concept-based modeling (GCP) approach, introducing a graphical criterion and the maximal T-Admissible set to identify architectures that support counterfactual reasoning. A closed-form procedure is provided for evaluating counterfactuals when a model is causally interpretable, and the authors prove a fundamental trade-off: increasing predictive capacity by enlarging the predictor set ${\mathbf{T}}$ reduces the set of admissible counterfactual queries. Empirical validation on BarMNIST and CelebA demonstrates that using non-descendant features yields consistent counterfactual answers, while blindly expanding feature usage can jeopardize interpretability; overall, the work guides the construction of interpretable models that can meaningfully respond to counterfactual questions.

Abstract

Understanding the predictions made by deep learning models remains a central challenge, especially in high-stakes applications. A promising approach is to equip models with the ability to answer counterfactual questions -- hypothetical ``what if?'' scenarios that go beyond the observed data and provide insight into a model reasoning. In this work, we introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models and observational data. We analyze two common model classes -- blackbox and concept-based predictors -- and show that neither is causally interpretable in general. To address this gap, we develop a framework for building models that are causally interpretable by design. Specifically, we derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query. This leads to a fundamental tradeoff between causal interpretability and predictive accuracy, which we characterize by identifying the unique maximal set of features that yields an interpretable model with maximal predictive expressiveness. Experiments corroborate the theoretical findings.

Figures (11)

• Figure 1: (a) Illustration of different model classes: counterfactually consistent models (blue) and blackbox/concept-based models (yellow). (b) Original input image and corresponding predictions from each model. (c) Counterfactual predictions: models in the top row predict consistently across instantiations within the class, while those in the bottom row produce inconsistent predictions.
• Figure 2: Causal diagrams for different types of predictive models.
• Figure 3: (Left) $\Omega'$ is causally interpretable if a query can be uniquely computed from the observational data. (Right) A query cannot be uniquely computed from the observational data if $\Omega'$ is not causally interpretable.
• Figure 4: (Left) As we want a model to answer more counterfactual queries (${\mathbf{W}}_\star^1\subseteq {\mathbf{W}}_\star^2$), the predictive power would decrease ($\text{Max-T-Ad}({\mathbf{W}}_\star^2)\subseteq \text{Max-T-Ad}({\mathbf{W}}_\star^1)$). (Right) As the predictive power increases (${\mathbf{T}}_1\subseteq {\mathbf{T}}_2$), interpretable counterfactuals would decrease ($\text{W-Ad}({\mathbf{T}}_2)\subseteq \text{W-Ad}({\mathbf{T}}_1)$).
• Figure 5: (a) Example images of BarMNIST dataset. (b) Causal diagram of GCP models. Red arrows represent the possible usage for predicting the label. (c) Causal interpretability-accuracy trade-off.

Theorems & Definitions (32)

• Definition 1: Augmented SCM
• Example 1: Blackbox Model
• Example 2: Concept-based Model
• Definition 2: Causal Interpretability
• Example 3: Continued from \ref{['ex:bp']}
• Proposition 1: Non-interpretability of BP
• Example 4: Continued from \ref{['ex:cp']}
• Definition 3: Generalized Concept-based Prediction
• Theorem 1: Graphical Criterion
• Definition 4: T-Admissible Set