Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problems

Yuheng Lei, Yao Lyu, Guojian Zhan, Tao Zhang, Jiangtao Li, Jianyu Chen, Shengbo Eben Li, Sifa Zheng

TL;DR

ZOAC introduces Zeroth-Order Actor-Critic, a framework that blends evolutionary search with an actor-critic architecture to address sequential decision problems without requiring differentiable policies. It replaces episode-level parameter mutation with step-wise exploration and derives an unbiased zeroth-order policy gradient, augmented by a critic and advantage estimation for variance reduction. The approach achieves superior sample efficiency and stability on a challenging rule-based autonomous driving task and remains competitive with gradient-based RL on standard Gymnasium benchmarks, despite using non-differentiable policy structures. This scalable, parallelizable method broadens the applicability of derivative-free optimization to SDPs in robotics and autonomous systems.

Abstract

Evolutionary algorithms (EAs) have shown promise in solving sequential decision problems (SDPs) by simplifying them to static optimization problems and searching for the optimal policy parameters in a zeroth-order way. While these methods are highly versatile, they often suffer from high sample complexity due to their ignorance of the underlying temporal structures. In contrast, reinforcement learning (RL) methods typically formulate SDPs as Markov Decision Process (MDP). Although more sample efficient than EAs, RL methods are restricted to differentiable policies and prone to getting stuck in local optima. To address these issues, we propose a novel evolutionary framework Zeroth-Order Actor-Critic (ZOAC). We propose to use step-wise exploration in parameter space and theoretically derive the zeroth-order policy gradient. We further utilize the actor-critic architecture to effectively leverage the Markov property of SDPs and reduce the variance of gradient estimators. In each iteration, ZOAC employs samplers to collect trajectories with parameter space exploration, and alternates between first-order policy evaluation (PEV) and zeroth-order policy improvement (PIM). To evaluate the effectiveness of ZOAC, we apply it to a challenging multi-lane driving task, optimizing the parameters in a rule-based, non-differentiable driving policy that consists of three sub-modules: behavior selection, path planning, and trajectory tracking. We also compare it with gradient-based RL methods on three Gymnasium tasks, optimizing neural network policies with thousands of parameters. Experimental results demonstrate the strong capability of ZOAC in solving SDPs. ZOAC significantly outperforms EAs that treat the problem as static optimization and matches the performance of gradient-based RL methods even without first-order information, in terms of total average return across all tasks.

Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problems

TL;DR

ZOAC introduces Zeroth-Order Actor-Critic, a framework that blends evolutionary search with an actor-critic architecture to address sequential decision problems without requiring differentiable policies. It replaces episode-level parameter mutation with step-wise exploration and derives an unbiased zeroth-order policy gradient, augmented by a critic and advantage estimation for variance reduction. The approach achieves superior sample efficiency and stability on a challenging rule-based autonomous driving task and remains competitive with gradient-based RL on standard Gymnasium benchmarks, despite using non-differentiable policy structures. This scalable, parallelizable method broadens the applicability of derivative-free optimization to SDPs in robotics and autonomous systems.

Abstract

Evolutionary algorithms (EAs) have shown promise in solving sequential decision problems (SDPs) by simplifying them to static optimization problems and searching for the optimal policy parameters in a zeroth-order way. While these methods are highly versatile, they often suffer from high sample complexity due to their ignorance of the underlying temporal structures. In contrast, reinforcement learning (RL) methods typically formulate SDPs as Markov Decision Process (MDP). Although more sample efficient than EAs, RL methods are restricted to differentiable policies and prone to getting stuck in local optima. To address these issues, we propose a novel evolutionary framework Zeroth-Order Actor-Critic (ZOAC). We propose to use step-wise exploration in parameter space and theoretically derive the zeroth-order policy gradient. We further utilize the actor-critic architecture to effectively leverage the Markov property of SDPs and reduce the variance of gradient estimators. In each iteration, ZOAC employs samplers to collect trajectories with parameter space exploration, and alternates between first-order policy evaluation (PEV) and zeroth-order policy improvement (PIM). To evaluate the effectiveness of ZOAC, we apply it to a challenging multi-lane driving task, optimizing the parameters in a rule-based, non-differentiable driving policy that consists of three sub-modules: behavior selection, path planning, and trajectory tracking. We also compare it with gradient-based RL methods on three Gymnasium tasks, optimizing neural network policies with thousands of parameters. Experimental results demonstrate the strong capability of ZOAC in solving SDPs. ZOAC significantly outperforms EAs that treat the problem as static optimization and matches the performance of gradient-based RL methods even without first-order information, in terms of total average return across all tasks.
Paper Structure (30 sections, 3 theorems, 43 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 30 sections, 3 theorems, 43 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Notations
  3. Sequential Decision Problems as Markov Decision Process
  4. Reinforcement Learning and Actor-Critic Methods
  5. Evolution Strategies
  6. Proposed Framework
  7. Step-wise Exploration in Parameter Space
  8. Zeroth-order Policy Gradient
  9. Optimization Objective
  10. Gradient Derivation
  11. Variance Reduction via Advantage Estimation
  12. Convergence Property
  13. Practical Zeroth-order Actor-critic Algorithm
  14. Overall Framework
  15. Real-valued Policy Encoding
  16. ...and 15 more sections

Key Result

Theorem 1

For any Markov Decision Process, the zeroth-order policy gradient of the optimization objective eq:method-zoacobj-avg or eq:method-zoacobj-dis with respect to the parameters of the Gaussian distribution is

Figures (9)

  • Figure 1: Existing approaches for sequential decision problems: (a) Evolutionary algorithms (EA), (b) Reinforcement Learning (RL). A crucial difference is EA formulates a static optimization problem while RL formulates an MDP with temporal credit assignment. (c) The proposed framework ZOAC, in which samplers collect rollouts with step-wise parameter exploration and the learner alternates between first-order policy evaluation and zeroth-order policy improvement.
  • Figure 2: Comparison of exploration strategies during sampling.
  • Figure 3: Task description of rule-based autonomous driving policy training.
  • Figure 4: State of ego vehicle and surrounding vehicles.
  • Figure 5: Learning curves of ZOAC and baseline methods in the autonomous driving task: (a) The solid lines represent the mean, and the shaded regions indicate the 95% confidence interval over 10 random seeds. All learning curves in this paper are presented in this manner unless otherwise specified. (b) Each solid line represents an independent training run, with runs that have a final performance not exceeding 4000 set to translucent for visual clarity.
  • ...and 4 more figures

Theorems & Definitions (6)

  • Theorem 1: Zeroth-order policy gradient
  • proof
  • Theorem 2: Variance upper bounds of gradient estimators
  • proof
  • Theorem 3: Convergence of ZOAC
  • proof