Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control

Hazim Alzorgan, Abolfazl Razi

TL;DR

MCBS addresses exploration challenges in continuous-action RL by augmenting TD3 with a structured look-ahead that samples $B$ candidate actions around the policy output and evaluates them with $D$-step Monte Carlo rollouts bootstrapped by the critic. The method selects the action with the highest estimated return and continues standard TD3 updates, with per-step complexity that scales as $O(B \cdot N_{\text{sim}} \cdot D)$. Empirical results on HalfCheetah-v4, Walker2d-v5, and Swimmer-v5 show faster convergence and higher final returns than TD3, while remaining competitive with SAC, PPO, and A2C; ablations highlight the importance of beam width $B$ and rollout depth $D$, and an adaptive rollout frequency reduces overhead. These findings support MCBS as a practical enhancement for sample-efficient continuous-control learning and motivate future extensions to model-based planning and real-time robotics.

Abstract

Actor-critic methods, like Twin Delayed Deep Deterministic Policy Gradient (TD3), depend on basic noise-based exploration, which can result in less than optimal policy convergence. In this study, we introduce Monte Carlo Beam Search (MCBS), a new hybrid method that combines beam search and Monte Carlo rollouts with TD3 to improve exploration and action selection. MCBS produces several candidate actions around the policy's output and assesses them through short-horizon rollouts, enabling the agent to make better-informed choices. We test MCBS across various continuous-control benchmarks, including HalfCheetah-v4, Walker2d-v5, and Swimmer-v5, showing enhanced sample efficiency and performance compared to standard TD3 and other baseline methods like SAC, PPO, and A2C. Our findings emphasize MCBS's capability to enhance policy learning through structured look-ahead search while ensuring computational efficiency. Additionally, we offer a detailed analysis of crucial hyperparameters, such as beam width and rollout depth, and explore adaptive strategies to optimize MCBS for complex control tasks. Our method shows a higher convergence rate across different environments compared to TD3, SAC, PPO, and A2C. For instance, we achieved 90% of the maximum achievable reward within around 200 thousand timesteps compared to 400 thousand timesteps for the second-best method.

Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control

TL;DR

MCBS addresses exploration challenges in continuous-action RL by augmenting TD3 with a structured look-ahead that samples candidate actions around the policy output and evaluates them with -step Monte Carlo rollouts bootstrapped by the critic. The method selects the action with the highest estimated return and continues standard TD3 updates, with per-step complexity that scales as . Empirical results on HalfCheetah-v4, Walker2d-v5, and Swimmer-v5 show faster convergence and higher final returns than TD3, while remaining competitive with SAC, PPO, and A2C; ablations highlight the importance of beam width and rollout depth , and an adaptive rollout frequency reduces overhead. These findings support MCBS as a practical enhancement for sample-efficient continuous-control learning and motivate future extensions to model-based planning and real-time robotics.

Abstract

Actor-critic methods, like Twin Delayed Deep Deterministic Policy Gradient (TD3), depend on basic noise-based exploration, which can result in less than optimal policy convergence. In this study, we introduce Monte Carlo Beam Search (MCBS), a new hybrid method that combines beam search and Monte Carlo rollouts with TD3 to improve exploration and action selection. MCBS produces several candidate actions around the policy's output and assesses them through short-horizon rollouts, enabling the agent to make better-informed choices. We test MCBS across various continuous-control benchmarks, including HalfCheetah-v4, Walker2d-v5, and Swimmer-v5, showing enhanced sample efficiency and performance compared to standard TD3 and other baseline methods like SAC, PPO, and A2C. Our findings emphasize MCBS's capability to enhance policy learning through structured look-ahead search while ensuring computational efficiency. Additionally, we offer a detailed analysis of crucial hyperparameters, such as beam width and rollout depth, and explore adaptive strategies to optimize MCBS for complex control tasks. Our method shows a higher convergence rate across different environments compared to TD3, SAC, PPO, and A2C. For instance, we achieved 90% of the maximum achievable reward within around 200 thousand timesteps compared to 400 thousand timesteps for the second-best method.
Paper Structure (32 sections, 4 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 32 sections, 4 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background and Related Work
  4. Reinforcement Learning Formulation
  5. Actor-Critic Methods
  6. Deterministic Policy Gradient (DPG)
  7. Twin Delayed Deep Deterministic Policy Gradient (TD3)
  8. MCTS
  9. Overview of MCTS in Discrete Domains
  10. Beam Search in Planning and RL
  11. Bridging TD3 with Look-Ahead Methods
  12. Proposed Method: MCBS for TD3
  13. MCBS Integration
  14. Advantages of MCBS for Continuous Control
  15. TD3 Foundation
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Overview of the Monte Carlo Beam Search enhanced TD3 algorithm. MCBS replaces standard action selection with a structured beam search and Monte Carlo rollouts, improving exploration and policy refinement.
  • Figure 2: Learning curves for HalfCheetah-v4, Walker2D-v5, and Swimmer-v5, comparing MCBS-TD3, A2C, PPO, TD3, and SAC over 1 million timesteps.
  • Figure 3: Comparison of MCBS-TD3 and standard TD3 across environments. MCBS significantly accelerates learning early on, but reducing rollout frequency after stabilization is crucial for efficiency.