Causal Entropy and Information Gain for Measuring Causal Control
Francisco Nunes Ferreira Quialheiro Simoes, Mehdi Dastani, Thijs van Ommen
TL;DR
The paper tackles the challenge of causal interpretability by extending entropy-based information theory to the interventional setting. It introduces causal entropy $H_c(Y\,|\, \\text{do}(X\sim X'))$ and causal information gain $I_c(Y\,|\, \\text{do}(X\sim X'))$, defined with intervention protocols, to quantify how much control a feature $X$ can exert over an outcome $Y$ and to relate this control to total causal effects. A key contribution is the formalization of these measures and the demonstration that they can correctly identify causal control in a structured example where standard mutual information misleads due to confounding. The work establishes a theoretical foundation for causally interpretable feature selection and motivates future estimators, extensions to continuous variables, and practical applications in domains with known or learnable causal structure.
Abstract
Artificial intelligence models and methods commonly lack causal interpretability. Despite the advancements in interpretable machine learning (IML) methods, they frequently assign importance to features which lack causal influence on the outcome variable. Selecting causally relevant features among those identified as relevant by these methods, or even before model training, would offer a solution. Feature selection methods utilizing information theoretical quantities have been successful in identifying statistically relevant features. However, the information theoretical quantities they are based on do not incorporate causality, rendering them unsuitable for such scenarios. To address this challenge, this article proposes information theoretical quantities that incorporate the causal structure of the system, which can be used to evaluate causal importance of features for some given outcome variable. Specifically, we introduce causal versions of entropy and mutual information, termed causal entropy and causal information gain, which are designed to assess how much control a feature provides over the outcome variable. These newly defined quantities capture changes in the entropy of a variable resulting from interventions on other variables. Fundamental results connecting these quantities to the existence of causal effects are derived. The use of causal information gain in feature selection is demonstrated, highlighting its superiority over standard mutual information in revealing which features provide control over a chosen outcome variable. Our investigation paves the way for the development of methods with improved interpretability in domains involving causation.