Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confounding Robust Deep Reinforcement Learning: A Causal Approach

Mingxuan Li, Junzhe Zhang, Elias Bareinboim

TL;DR

This work addresses off-policy reinforcement learning in the presence of unobserved confounding by framing the problem as a Confounded Markov Decision Process and applying partial identification to derive a causal lower bound on the optimal value function. The authors extend the Bellman equation to a causal form and introduce Causal-DQN, which estimates a bound on $Q_*(s,x)$ using nominal transitions and rewards derived from observational data and conservative next-state updates when the chosen action differs from the observed one. Empirically, Causal-DQN outperforms standard DQN baselines across 12 confounded Atari games, often matching or exceeding demonstrator performance under masked observations and showing improved sample efficiency. The results underline the importance of causal reasoning in RL when data are confounded, with implications for robust offline RL, RLHF, and real-world robotics, while outlining avenues for future work in extending the approach to policy-gradient methods, continuous control, and multi-agent settings. $Q_*(s,x)$ is bounded below by a computable $oxed{ ext{underline}{Q_*}(s,x)}$ that integrates observational estimates, action probabilities, and worst-case next states, enabling safe policy learning under confounding.

Abstract

A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where \emph{unobserved confounding} cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.

Confounding Robust Deep Reinforcement Learning: A Causal Approach

TL;DR

This work addresses off-policy reinforcement learning in the presence of unobserved confounding by framing the problem as a Confounded Markov Decision Process and applying partial identification to derive a causal lower bound on the optimal value function. The authors extend the Bellman equation to a causal form and introduce Causal-DQN, which estimates a bound on using nominal transitions and rewards derived from observational data and conservative next-state updates when the chosen action differs from the observed one. Empirically, Causal-DQN outperforms standard DQN baselines across 12 confounded Atari games, often matching or exceeding demonstrator performance under masked observations and showing improved sample efficiency. The results underline the importance of causal reasoning in RL when data are confounded, with implications for robust offline RL, RLHF, and real-world robotics, while outlining avenues for future work in extending the approach to policy-gradient methods, continuous control, and multi-agent settings. is bounded below by a computable that integrates observational estimates, action probabilities, and worst-case next states, enabling safe policy learning under confounding.

Abstract

A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where \emph{unobserved confounding} cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.
Paper Structure (16 sections, 3 theorems, 20 equations, 12 figures, 2 tables, 2 algorithms)

This paper contains 16 sections, 3 theorems, 20 equations, 12 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Challenges Due to Unobserved Confounders
  3. Off-Policy Learning.
  4. Confounding Robust Deep Q-Learning
  5. Experiments
  6. Confounded Boxing.
  7. Confounded Gopher.
  8. Remark on the failure of DQN baselines.
  9. Conclusions
  10. Related Work
  11. Proof Details
  12. Confounded Atari Games Design
  13. Implementation Details and Experiment Setups
  14. More Experiment Results
  15. Broader Impact
  16. ...and 1 more sections

Key Result

Proposition 3.1

For a CMDP environment $\mathcal{M}$ with reward signals $Y_t \in [a, b] \subseteq \mathbb{R}$, its optimal state-action value function $Q_{*}(s, x) \geq \underline{Q_{*}}(s, x)$ for any state-action pair $(s, x) \in \mathcal{S} \times \mathcal{X}$, where the lower bound $\underline{Q_{*}}(s, x)$ is where $P(x \mid s) = P\left(X_t = x \mid S_t = s\right)$ and $P(\neg x \mid s) = 1 - P(x \mid s)$;

Figures (12)

  • Figure 1: Causal diagram representing the data-generating mechanisms in a Confounded Markov Decision Process.
  • Figure 2: (\ref{['fig:_2_2_a']}) A saliency map of the behavioral policy in Pong that tracks the opponent's location and score board; (\ref{['fig:_2_2_b']}) a confounded Pong game where the opponent's location and score board is masked; (\ref{['fig:_2_2_c']}) a saliency map of the conservative policy focusing on only itself and the ball; (\ref{['fig:_2_2_d']}) the average return of our causal DQN and the standard DQN baselines. Baseline curves are overlapped.
  • Figure 3: Backup diagram for causal deep Q-learning.
  • Figure 4: (\ref{['fig:_4_4_a']}) A saliency map of the demonstrator's policy; (\ref{['fig:_4_4_b']}) a confounded Boxing game where only the left half of the arena is visible; (\ref{['fig:_4_4_c']}) a saliency map of Causal-DQN's policy.
  • Figure 5: (\ref{['fig:_4_3_a']}) A saliency map of the demonstrator's policy; (\ref{['fig:_4_3_b']}) a confounded Gopher game where the tunnel and score are masked; (\ref{['fig:_4_3_c']}) a saliency map of Causal-DQN's policy.
  • ...and 7 more figures

Theorems & Definitions (8)

  • Definition 2.1
  • Example 1: Confounded Pong
  • Proposition 3.1: Causal Bellman Optimality Equation
  • Example 2: Confounded Pong continued
  • Proposition B.1: Causal Bellman Optimal Equation (\ref{['prop:_3_1']})
  • proof
  • Proposition B.2: Convergence of Causal Bellman Optimal Equation in Stationary CMDPs
  • proof