Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning

Yukinari Hisaki, Isao Ono

TL;DR

RVI-SAC tackles off-policy average-reward reinforcement learning for continuous control by fusing RVI Q-learning with a deep actor–critic framework. It introduces Delayed f(Q) Update to stabilize neural Q-network targets, and proves a convergence result akin to the tabular case, while deriving an Average Reward Soft Policy Improvement theorem to safely update the actor under the average-reward objective. The method includes Automatic Reset Cost adjustment to handle terminated tasks, enabling application to locomotion problems like Mujoco. Empirically, RVI-SAC achieves competitive performance against SAC with various discount rates and ARO-DDPG, demonstrating robustness to discount choices and improved sample efficiency under the average-reward criterion.

Abstract

In this paper, we propose an off-policy deep reinforcement learning (DRL) method utilizing the average reward criterion. While most existing DRL methods employ the discounted reward criterion, this can potentially lead to a discrepancy between the training objective and performance metrics in continuing tasks, making the average reward criterion a recommended alternative. We introduce RVI-SAC, an extension of the state-of-the-art off-policy DRL method, Soft Actor-Critic (SAC), to the average reward criterion. Our proposal consists of (1) Critic updates based on RVI Q-learning, (2) Actor updates introduced by the average reward soft policy improvement theorem, and (3) automatic adjustment of Reset Cost enabling the average reward reinforcement learning to be applied to tasks with termination. We apply our method to the Gymnasium's Mujoco tasks, a subset of locomotion tasks, and demonstrate that RVI-SAC shows competitive performance compared to existing methods.

RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning

TL;DR

RVI-SAC tackles off-policy average-reward reinforcement learning for continuous control by fusing RVI Q-learning with a deep actor–critic framework. It introduces Delayed f(Q) Update to stabilize neural Q-network targets, and proves a convergence result akin to the tabular case, while deriving an Average Reward Soft Policy Improvement theorem to safely update the actor under the average-reward objective. The method includes Automatic Reset Cost adjustment to handle terminated tasks, enabling application to locomotion problems like Mujoco. Empirically, RVI-SAC achieves competitive performance against SAC with various discount rates and ARO-DDPG, demonstrating robustness to discount choices and improved sample efficiency under the average-reward criterion.

Abstract

In this paper, we propose an off-policy deep reinforcement learning (DRL) method utilizing the average reward criterion. While most existing DRL methods employ the discounted reward criterion, this can potentially lead to a discrepancy between the training objective and performance metrics in continuing tasks, making the average reward criterion a recommended alternative. We introduce RVI-SAC, an extension of the state-of-the-art off-policy DRL method, Soft Actor-Critic (SAC), to the average reward criterion. Our proposal consists of (1) Critic updates based on RVI Q-learning, (2) Actor updates introduced by the average reward soft policy improvement theorem, and (3) automatic adjustment of Reset Cost enabling the average reward reinforcement learning to be applied to tasks with termination. We apply our method to the Gymnasium's Mujoco tasks, a subset of locomotion tasks, and demonstrate that RVI-SAC shows competitive performance compared to existing methods.
Paper Structure (28 sections, 6 theorems, 76 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 28 sections, 6 theorems, 76 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Markov Decision Process
  4. Average Reward Reinforcement Learning
  5. RVI Q-learning
  6. Proposed Method
  7. RVI Q-learning based Q-Network update
  8. Average Reward Soft Policy Improvement Theorem
  9. Automatic Reset Cost Adjustment
  10. Experiment
  11. Comparative evaluation
  12. Design evaluation
  13. Related Works
  14. Conclusion
  15. Mathematical Notations
  16. ...and 13 more sections

Key Result

Theorem 3.1

The algorithm expressed by the following equations converges almost surely to a uniquely determined $q^*$ under appropriate assumptions (see Appendix sec:convergence_proof_of_RVI_Q_learning_with_Delayed_fQ_Update):

Figures (4)

  • Figure 1: Learning curves for the Gymnasium's Mujoco tasks. The horizontal axis represents Steps, and the vertical axis represents the evaluation value (total_return). Lines and shades represent the mean and standard deviation of the evaluation values over 10 trials, respectively.
  • Figure 2: Experimental results demonstrating the effectiveness of each component of RVI-SAC. All three graphs represent learning curves on the Ant environment.
  • Figure 3: Learning curves for the Gymnasium's Mujoco tasks. The horizontal axis represents Steps, and the vertical axis represents the evaluation value (average reward). Lines and shades represent the mean and standard deviation of the evaluation values over 10 trials, respectively.
  • Figure 4: This figure represent learning curves for all environments with termination, compare RVI-SAC (red) with SAC (blue) with automatic Reset Cost adjustment and ARO-DDPG(purple) with automatic Reset Cost adjustment.

Theorems & Definitions (9)

  • Theorem 3.1: Sketch
  • Theorem 3.2: Average Reward Soft Policy Improvement
  • proof
  • Theorem 3.10
  • Lemma 3.12
  • Lemma 3.13
  • proof
  • Lemma 3.14
  • proof