Why Online Reinforcement Learning is Causal

TL;DR

The paper investigates when causal modeling adds value to reinforcement learning by distinguishing online and offline settings. It argues that online RL yields inherently causal probabilities because agents learn from their own actions and observations, aligning conditional probabilities with interventional effects; offline RL, especially under partial observability, can exhibit confounding that requires causal models and counterfactual reasoning. Using causal Bayesian networks, SCMs, and dynamic decision networks, the authors formalize how interventions, what-if counterfactuals, and hindsight counterfactuals can be computed and related to traditional RL quantities, establishing conditions (notably action sufficiency and observation-equivalence in online learning) under which conditional and interventional probabilities coincide. The work provides a structured framework to analyze online/offline RL through a four-level causal hierarchy, demonstrates how to perform model-based evaluation with DDNs, and discusses future directions such as causal OPE, hindsight augmentation, and learning causal models from data. Overall, the results clarify when causal reasoning improves RL and outline practical steps for integrating causal models into online and offline RL pipelines to address distribution shift and counterfactual reasoning in decision making.

Abstract

Reinforcement learning (RL) and causal modelling naturally complement each other. The goal of causal modelling is to predict the effects of interventions in an environment, while the goal of reinforcement learning is to select interventions that maximize the rewards the agent receives from the environment. Reinforcement learning includes the two most powerful sources of information for estimating causal relationships: temporal ordering and the ability to act on an environment. This paper examines which reinforcement learning settings we can expect to benefit from causal modelling, and how. In online learning, the agent has the ability to interact directly with their environment, and learn from exploring it. Our main argument is that in online learning, conditional probabilities are causal, and therefore offline RL is the setting where causal learning has the most potential to make a difference. Essentially, the reason is that when an agent learns from their {\em own} experience, there are no unobserved confounders that influence both the agent's own exploratory actions and the rewards they receive. Our paper formalizes this argument. For offline RL, where an agent may and typically does learn from the experience of {\em others}, we describe previous and new methods for leveraging a causal model, including support for counterfactual queries.

Figures (20)

• Figure 1: Dynamic influence diagrams for a generic reward model. We follow the conventions of influence diagrams to distinguish state variables, actions, and rewards. Observed variables are gray, latent variables white. \ref{['fig:temporal']}: States and actions temporally precede rewards. Therefore rewards to not cause states/actions, and reward probabilities are causal (i.e., $P(R_{t+1}|\mathit{do}(A_t = a_t),S_t) = P(R_{t+1}|A_t = a_t,S_t)$), unless there is an unobserved confounder. \ref{['fig:confounded']}: The environment state $S_t = (O_t,Z_t)$ comprises an observation signal $O_t$ and a latent part $Z_t$. The unobserved variable $Z_t$ is a latent common cause (confounder) of both actions and rewards. Because of the confounder, conditional probabilities generally do not correctly estimate the causal effects of actions (i.e., $P(R_{t+1}|\mathit{do}(A_t = a_t),S_t) \neq P(R_{t+1}|A_t = a_t,S_t)$).
• Figure 2: Online RL settings; Figures adapted from Levine2020. \ref{['fig:levine-online']}: In classic online RL, the policy $\pi_{k}$ is updated with streaming data collected by $\pi_{k}$ itself. \ref{['fig:levine-offpolicy']}: In classic off-policy RL, the agent’s online experience is appended to a data buffer (also called a replay buffer) $D$, each new policy $\pi_{k}$ collects additional data, such that $D$ comprises samples from $\pi_{0},\ldots,\pi_{k}$, and all of this data is used to train an updated new policy $\pi_{k+1}$. Both online settings satisfy observation-equivalence because the policies used to generate the data are based on the same observations as the learned policy.
• Figure 3: Offline RL-learning employs a dataset $D$ collected by some (potentially unknown) behavior policy $\pi_{\beta}$. The dataset is collected once, and is not altered during training. The training process does not interact with the environment directly, and the policy is only deployed after being fully trained. \ref{['fig:levine-offline']}Levine2020: offline RL with observation-equivalence where the behavioral policy $\pi_{\beta}$ and the learned policy $\pi$ are based on the same observation signal. \ref{['fig:partial-offline']}: offline RL without observation-equivalence where the behavior policy $\pi_{\beta}$ has access to more observations than the learned policy $\pi$.
• Figure 4: Causal Graphs for a sports scenario like hockey or soccer. We follow the conventions of influence diagrams to distinguish state variables, actions, and rewards. All variables are binary. Variables observed by the learning agent are gray, latent variables white. Whether a player takes a shot depends on their location and whether they are injured. Likewise, the chance of their shot leading to a goal depends on their location and health. Thus Player Health is a common cause of the action and reward. \ref{['fig:first']}: In the online setting, the athlete learns from their own experience, which includes their health. \ref{['fig:confound']}: In the offline setting, the learner is different from the athlete, for example a coach, and does not observe the health of the behavioral agent. Player health is an unobserved confounder of action and reward.
• Figure 5: Reinforcement learning settings in which we can expect conditional probabilities to be equivalent to interventional probabilities.

Theorems & Definitions (18)

• Lemma 1
• Lemma 2
• proof
• Definition 1
• proof
• Proposition 1
• proof
• Proposition 2
• proof
• Lemma 3