Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RN-D: Discretized Categorical Actors with Regularized Networks for On-Policy Reinforcement Learning

Yuexin Bian, Jie Feng, Tao Wang, Yijiang Li, Sicun Gao, Yuanyuan Shi

TL;DR

RN-D introduces discretized categorical actors with regularized networks for on-policy reinforcement learning, reframing policy optimization as cross-entropy-like learning over per-dimension action bins. The approach replaces diagonal Gaussian actors with a discretized policy and pairs it with a Regularized Actor Network architecture featuring residual blocks and pre-normalization, resulting in lower gradient variance and faster convergence. Empirical results across MuJoCo locomotion and ManiSkill manipulation tasks show state-of-the-art performance and improved sample efficiency, including vision-based settings. This work highlights policy representation and actor architecture as powerful levers for robust, scalable on-policy control, and suggests broad applicability to other on-policy algorithms and complex tasks.

Abstract

On-policy deep reinforcement learning remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy and policy updates must be conservative. In this paper, we revisit policy representation as a first-class design choice for on-policy optimization. We study discretized categorical actors that represent each action dimension with a distribution over bins, yielding a policy objective that resembles a cross-entropy loss. Building on architectural advances from supervised learning, we further propose regularized actor networks, while keeping critic design fixed. Our results show that simply replacing the standard actor network with our discretized regularized actor yields consistent gains and achieve the state-of-the-art performance across diverse continuous-control benchmarks.

RN-D: Discretized Categorical Actors with Regularized Networks for On-Policy Reinforcement Learning

TL;DR

RN-D introduces discretized categorical actors with regularized networks for on-policy reinforcement learning, reframing policy optimization as cross-entropy-like learning over per-dimension action bins. The approach replaces diagonal Gaussian actors with a discretized policy and pairs it with a Regularized Actor Network architecture featuring residual blocks and pre-normalization, resulting in lower gradient variance and faster convergence. Empirical results across MuJoCo locomotion and ManiSkill manipulation tasks show state-of-the-art performance and improved sample efficiency, including vision-based settings. This work highlights policy representation and actor architecture as powerful levers for robust, scalable on-policy control, and suggests broad applicability to other on-policy algorithms and complex tasks.

Abstract

On-policy deep reinforcement learning remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy and policy updates must be conservative. In this paper, we revisit policy representation as a first-class design choice for on-policy optimization. We study discretized categorical actors that represent each action dimension with a distribution over bins, yielding a policy objective that resembles a cross-entropy loss. Building on architectural advances from supervised learning, we further propose regularized actor networks, while keeping critic design fixed. Our results show that simply replacing the standard actor network with our discretized regularized actor yields consistent gains and achieve the state-of-the-art performance across diverse continuous-control benchmarks.
Paper Structure (51 sections, 1 theorem, 28 equations, 8 figures, 2 tables)

This paper contains 51 sections, 1 theorem, 28 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. On-policy reinforcement learning.
  4. Actor policy parameterization for continuous control.
  5. Network architectures.
  6. Preliminaries
  7. On-policy actor optimization
  8. Policy parameterizations for continuous control
  9. Continuous Gaussian actor.
  10. Discrete categorical actor via uniform discretization.
  11. Training with PPO.
  12. Revisiting Discrete Categorical Policies
  13. Policy-gradient estimator and variance
  14. On-policy policy gradient.
  15. A vanilla policy-gradient toy setting.
  16. ...and 36 more sections

Key Result

Proposition 4.1

Fix a state $s$ and consider the one-step REINFORCE estimator $\hat{g}(\theta)=R\,\nabla_\theta \log \pi_\theta(a\mid s)$ with $a\sim\pi_\theta(\cdot\mid s)$ and constant return $R$. Then the conditional covariance for the two policy families are: [1] Gaussian (gradient w.r.t. mean). Denote $\pi^c(a [2] Categorical (gradient w.r.t. logits). Denote the discretized categorical policy factorize acros

Figures (8)

  • Figure 1: Understanding Continuous Gaussian and Discrete categorical actor policy from two perspective.
  • Figure 2: Proposed Regularized Network for Discrete action policies (RN-D). The actor consists of a feature extractor (MLP or CNN) followed by pre-LayerNorm residual MLP blocks, enabling stable optimization and improved scalability. The output layer produces categorical logits over $K$ bins for each action dimension.
  • Figure 3: Aggregate learning curves across benchmarks. Each subplot reports the mean performance over tasks within a benchmark (MuJoCo locomotion: normalized return; ManiSkill: success rate) as a function of environment steps. Curves are averaged over 5 random seeds; shaded regions denote 95% stratified bootstrap confidence intervals. The red annotations indicate a sample-efficiency speedup.
  • Figure 4: The evolution of the policy-gradient variance over training (log scale).
  • Figure 5: Component Analysis. (a) Gradient signal-to-noise ratio (SNR) on Gym locomotion tasks. Bars show the mean and error bars denote one standard deviation across tasks. (b) Average normalized return. Higher SNR correlates with higher returns, with RN-D achieving the best performance.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Proposition 4.1: Gradient variance for Gaussian vs. categorical policies
  • proof
  • proof