Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Discovery of Functional Actual Causes in Complex Environments

Caleb Chuck, Sankaran Vaidyanathan, Stephen Giguere, Amy Zhang, David Jensen, Scott Niekum

TL;DR

This work tackles the challenge of causalgeneralization in reinforcement learning by introducing Functional Actual Cause (FAC), a principled framework that constrains actual causes using invariant preimages (IVP) to capture context-specific independencies and normalize causation judgments. It then presents Joint Optimization for Actual Cause Inference (JACI), a neural approach that jointly learns a state-to-cause mapping and a masked forward model to recover functional actual causes from observational data in continuous, high-dimensional environments. The approach is shown to align with established causality intuitions on classic examples and to outperform baselines in synthetic Random Vector domains and RL-like tasks such as Mini-Breakout and 2D Pushing, demonstrating scalable, accurate identification of sparse, context-relevant causes. Collectively, FAC and JACI offer a scalable bridge between formal actual causality and practical inference for RL, enabling improved world-modeling, explanations, and exploration in complex environments.

Abstract

Reinforcement learning (RL) algorithms often struggle to learn policies that generalize to novel situations due to issues such as causal confusion, overfitting to irrelevant factors, and failure to isolate control of state factors. These issues stem from a common source: a failure to accurately identify and exploit state-specific causal relationships in the environment. While some prior works in RL aim to identify these relationships explicitly, they rely on informal domain-specific heuristics such as spatial and temporal proximity. Actual causality offers a principled and general framework for determining the causes of particular events. However, existing definitions of actual cause often attribute causality to a large number of events, even if many of them rarely influence the outcome. Prior work on actual causality proposes normality as a solution to this problem, but its existing implementations are challenging to scale to complex and continuous-valued RL environments. This paper introduces functional actual cause (FAC), a framework that uses context-specific independencies in the environment to restrict the set of actual causes. We additionally introduce Joint Optimization for Actual Cause Inference (JACI), an algorithm that learns from observational data to infer functional actual causes. We demonstrate empirically that FAC agrees with known results on a suite of examples from the actual causality literature, and JACI identifies actual causes with significantly higher accuracy than existing heuristic methods in a set of complex, continuous-valued environments.

Automated Discovery of Functional Actual Causes in Complex Environments

TL;DR

This work tackles the challenge of causalgeneralization in reinforcement learning by introducing Functional Actual Cause (FAC), a principled framework that constrains actual causes using invariant preimages (IVP) to capture context-specific independencies and normalize causation judgments. It then presents Joint Optimization for Actual Cause Inference (JACI), a neural approach that jointly learns a state-to-cause mapping and a masked forward model to recover functional actual causes from observational data in continuous, high-dimensional environments. The approach is shown to align with established causality intuitions on classic examples and to outperform baselines in synthetic Random Vector domains and RL-like tasks such as Mini-Breakout and 2D Pushing, demonstrating scalable, accurate identification of sparse, context-relevant causes. Collectively, FAC and JACI offer a scalable bridge between formal actual causality and practical inference for RL, enabling improved world-modeling, explanations, and exploration in complex environments.

Abstract

Reinforcement learning (RL) algorithms often struggle to learn policies that generalize to novel situations due to issues such as causal confusion, overfitting to irrelevant factors, and failure to isolate control of state factors. These issues stem from a common source: a failure to accurately identify and exploit state-specific causal relationships in the environment. While some prior works in RL aim to identify these relationships explicitly, they rely on informal domain-specific heuristics such as spatial and temporal proximity. Actual causality offers a principled and general framework for determining the causes of particular events. However, existing definitions of actual cause often attribute causality to a large number of events, even if many of them rarely influence the outcome. Prior work on actual causality proposes normality as a solution to this problem, but its existing implementations are challenging to scale to complex and continuous-valued RL environments. This paper introduces functional actual cause (FAC), a framework that uses context-specific independencies in the environment to restrict the set of actual causes. We additionally introduce Joint Optimization for Actual Cause Inference (JACI), an algorithm that learns from observational data to infer functional actual causes. We demonstrate empirically that FAC agrees with known results on a suite of examples from the actual causality literature, and JACI identifies actual causes with significantly higher accuracy than existing heuristic methods in a set of complex, continuous-valued environments.
Paper Structure (52 sections, 4 theorems, 21 equations, 7 figures, 8 tables, 3 algorithms)

This paper contains 52 sections, 4 theorems, 21 equations, 7 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Illustrative Example
  3. Background and Related Work
  4. Functional Actual Cause
  5. Invariant Preimage
  6. Causal Sufficiency
  7. Causal Necessity
  8. Minimality
  9. Actual Causation
  10. FAC and Normality
  11. Relationships between FAC and existing definitions
  12. Discovering Actual Causes
  13. Recovering Binary-Subset Pairs by Joint Optimization
  14. Experiments
  15. Baselines
  16. ...and 37 more sections

Key Result

Theorem 4.6

For an observed state $\mathbf{s^*}$ and binary-subset pair $(\mathcal{B}(\mathbf{Y}), \mathcal{I}(\mathbf{Y}))$, if the event $\mathbf{X} = \mathbf{x}$ satisfies AC1, functional sufficiency, and contrastive necessity for the outcome $\mathbf{Y} = \mathbf{y}$ under witness set $\mathbf{W}$, and $(\m

Figures (7)

  • Figure 1: (a)Invariant preimages of the block position corresponding to a portion of the state space from Example \ref{['block-pushing-example']}. The block is at position 50 and the pusher attempts to move it to position 52. In states (a) and (c), the obstacle has no impact on the block position. In state (b) the obstacle is directly in front of the block, and in state (d) it is one unit away and will obstruct the pusher. The blue region represents states where only the pusher can affect the block position---an invariant preimage of the block position with respect to the pusher. When the observed state is in this region, the obstacle is not an actual cause of the block being pushed successfully.
  • Figure 2: (a)2D Pushing: the continuous actions move the pusher in $[-1,-1]^2$ in a $5$ by $5$ area. (b)Mini-Breakout: the actions move the paddle to the right and left.
  • Figure 3: (a) Computational flow of $h_\mathbf{Y}(\mathbf s;\theta)$, which takes in the observed state for all causal variables and returns a binary vector indicating the actual cause. (b) Computational flow of $f(\mathbf s, \mathbf b;\phi)$, which takes in the observed state for all causal variables and a binary vector and returns the state of the outcome variable.
  • Figure 4: Random Vectors domains, where dotted lines indicate conditional edges. In 3-m-in, the three edges form a single connection such that the outcome varies according to all of the inputs. $\tau$-1 and d-20 use the same graph structure as 1-in.
  • Figure 5: (a) Comparison of false positive and false negative rates in the 1-in domain. The blue line indicates false positive rate (the model guesses an actual cause when there is none) and the orange line indicates false negative rate (the model guesses no actual cause when there is one). (b) Train curve averaged over 10 runs for 1-in. Error rate decreases monotonically, even though JACI optimizes Equation \ref{['optimization-learned-model']} instead of the error rate explicitly.
  • ...and 2 more figures

Theorems & Definitions (18)

  • Example 2.1: 1-D Block Pushing
  • Definition 3.1: General Framework for Actual Causation
  • Definition 4.1: Invariant Preimage
  • Definition 4.2: Weak Sufficiency
  • Definition 4.3: Functional Sufficiency
  • Definition 4.4: Contrastive Necessity
  • Definition 4.5: Minimality of binary-subset pair
  • Theorem 4.6
  • Definition 4.7: Functional Actual Cause
  • Theorem 4.8
  • ...and 8 more