Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimation

Wenzhang Liu, Lianjun Jin, Lu Ren, Chaoxu Mu, Changyin Sun

TL;DR

This work tackles action redundancy in deep reinforcement learning by introducing Causal Effect Estimation (CEE), a two-phase framework that pre-trains an inverse dynamics model and an N-value network to efficiently compute per-action causal effects on one-step state transitions. Actions are then grouped via similarity and filtered through a relative-effect masking mechanism to form a Minimal Causal Action Space, guiding exploration with a masked policy. Theoretical guarantees link positive causal effect to a causal action via $C^{\pi}(A|S \rightarrow S') > 0$, and empirical results across Maze, MiniGrid, and Atari demonstrate improved learning efficiency and performance over PPO and NPM baselines, including in large action spaces. Overall, CEE provides a practical, generalizable method to reduce DRL action spaces without relying on extensive environment-specific priors, enhancing exploration and policy quality in complex decision problems. Key contributions include a causal graphical model for state transitions, a KL-divergence-based causal effect measure, an N-value–driven estimation procedure, and an action-classification–augmented masking strategy with robust empirical support.

Abstract

Intelligent decision-making within large and redundant action spaces remains challenging in deep reinforcement learning. Considering similar but ineffective actions at each step can lead to repetitive and unproductive trials. Existing methods attempt to improve agent exploration by reducing or penalizing redundant actions, yet they fail to provide quantitative and reliable evidence to determine redundancy. In this paper, we propose a method to improve exploration efficiency by estimating the causal effects of actions. Unlike prior methods, our approach offers quantitative results regarding the causality of actions for one-step transitions. We first pre-train an inverse dynamics model to serve as prior knowledge of the environment. Subsequently, we classify actions across the entire action space at each time step and estimate the causal effect of each action to suppress redundant actions during exploration. We provide a theoretical analysis to demonstrate the effectiveness of our method and present empirical results from simulations in environments with redundant actions to evaluate its performance. Our implementation is available at https://github.com/agi-brain/cee.git.

Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimation

TL;DR

This work tackles action redundancy in deep reinforcement learning by introducing Causal Effect Estimation (CEE), a two-phase framework that pre-trains an inverse dynamics model and an N-value network to efficiently compute per-action causal effects on one-step state transitions. Actions are then grouped via similarity and filtered through a relative-effect masking mechanism to form a Minimal Causal Action Space, guiding exploration with a masked policy. Theoretical guarantees link positive causal effect to a causal action via , and empirical results across Maze, MiniGrid, and Atari demonstrate improved learning efficiency and performance over PPO and NPM baselines, including in large action spaces. Overall, CEE provides a practical, generalizable method to reduce DRL action spaces without relying on extensive environment-specific priors, enhancing exploration and policy quality in complex decision problems. Key contributions include a causal graphical model for state transitions, a KL-divergence-based causal effect measure, an N-value–driven estimation procedure, and an action-classification–augmented masking strategy with robust empirical support.

Abstract

Intelligent decision-making within large and redundant action spaces remains challenging in deep reinforcement learning. Considering similar but ineffective actions at each step can lead to repetitive and unproductive trials. Existing methods attempt to improve agent exploration by reducing or penalizing redundant actions, yet they fail to provide quantitative and reliable evidence to determine redundancy. In this paper, we propose a method to improve exploration efficiency by estimating the causal effects of actions. Unlike prior methods, our approach offers quantitative results regarding the causality of actions for one-step transitions. We first pre-train an inverse dynamics model to serve as prior knowledge of the environment. Subsequently, we classify actions across the entire action space at each time step and estimate the causal effect of each action to suppress redundant actions during exploration. We provide a theoretical analysis to demonstrate the effectiveness of our method and present empirical results from simulations in environments with redundant actions to evaluate its performance. Our implementation is available at https://github.com/agi-brain/cee.git.
Paper Structure (27 sections, 4 theorems, 25 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 27 sections, 4 theorems, 25 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Eliminating Redundant Actions in DRL
  4. Estimating the Causal Effects of Actions
  5. Background
  6. Policy with Action Mask
  7. Causal Graphical Model for State Transition
  8. Method
  9. Causal Effects Estimation for Actions
  10. Estimating Causal Effects of Actions Using Modified Inverse Dynamic Model
  11. Actions Classification
  12. Minimal Causal Action Space
  13. Experiments
  14. Simulated Environments
  15. Maze
  16. ...and 12 more sections

Key Result

Theorem 1

Consider a causal graphical model $\mathcal{G}$ induced by a set of nodes $\mathcal{V}=\{S, \pi, A, S'\}$ and the set of edges $\mathcal{E}$. $A \in \mathcal{V}$ is a cause of $S'$ given $S$ (i.e. there is an edge from $A$ to $S'$), if and only if $C^{\pi}(A | S \rightarrow S') > 0$, otherwise, $C^{

Figures (9)

  • Figure 1: Causal graphical model for state transition.
  • Figure 2: The framework of our method.
  • Figure 3: Illustration of the Maze environment and the results: (a) Every actuator has two states: on and off. The agent's movement is determined by the vector sum of all the actuators that are on. The agent's task is to reach the goal; (b) Results on Maze environment with 64 actions; (c) Results on Maze environment with 128 actions.
  • Figure 4: Results on 8 different tasks in MiniGrid environment.
  • Figure 5: Results on part of tasks in Atari 2600 environment.
  • ...and 4 more figures

Theorems & Definitions (10)

  • Definition 1: Causal Actions
  • Definition 2: Causal Effects of Actions
  • Theorem 1
  • Definition 3: Approximate Causal Action Space
  • Lemma 1
  • Definition 4: Relative Causal Effects of Actions
  • Theorem 1
  • proof
  • Lemma 1
  • proof