An agent design with goal reaching guarantees for enhancement of learning

TL;DR

This work addresses reinforcement learning in MDPs with a designated goal region $\\mathbb{G}$ and a probabilistic goal-reaching property $\\Pi_0$. It introduces a critic-based augmentation that preserves the goal-reaching guarantee while accelerating learning: a state-valued critic is updated only when $\\hat{V}^w(s_{t+1}) - \\hat{V}^{w^\\dagger}(s_t^{\\dagger}) > 0$, with a relaxation mechanism $P_{relax}$ decaying at rate $\\lambda_{relax}$ and bounded critic outputs via $-\\hat{\\kappa}_{up}(\\|s\\|) \le \\hat{V}^w(s) \le -\\hat{\\kappa}_{low}(\\|s\\|)$. Theoretical guarantees (Theorem 1) show the policy sequence $\\pi_t$ retains the goal-reaching property with probability at least $1 - \\eta$, and experiments on six environments demonstrate faster learning while achieving final performance on par with or better than strong baselines such as PPO, DDPG, SAC, TD3, and REINFORCE. The approach can be layered on top of any critic-based agent, potentially improving sample efficiency without sacrificing the goal-reaching objective. The authors provide a formal proof and empirical evidence, with open-source code at the referenced repository.

Abstract

Reinforcement learning is commonly concerned with problems of maximizing accumulated rewards in Markov decision processes. Oftentimes, a certain goal state or a subset of the state space attain maximal reward. In such a case, the environment may be considered solved when the goal is reached. Whereas numerous techniques, learning or non-learning based, exist for solving environments, doing so optimally is the biggest challenge. Say, one may choose a reward rate which penalizes the action effort. Reinforcement learning is currently among the most actively developed frameworks for solving environments optimally by virtue of maximizing accumulated reward, in other words, returns. Yet, tuning agents is a notoriously hard task as reported in a series of works. Our aim here is to help the agent learn a near-optimal policy efficiently while ensuring a goal reaching property of some basis policy that merely solves the environment. We suggest an algorithm, which is fairly flexible, and can be used to augment practically any agent as long as it comprises of a critic. A formal proof of a goal reaching property is provided. Comparative experiments on several problems under popular baseline agents provided an empirical evidence that the learning can indeed be boosted while ensuring goal reaching property.

Figures (13)

• Figure 1: The plots show smoothed learning curves, representing accumulated episodic reward versus the number of environment steps. The plots represent median performance over 10 random seeds relative to the baseline policy $\pi_0$, with the accumulated reward of $\pi_0$ subtracted for clarity. Each plot is smoothed using a rolling median followed by Bezier interpolation. Plots are truncated starting when policy gradient algorithms (PPO, DDPG, VPG, REINFORCE, SAC, TD3) reach $\pi_0$ performance in value, while for our agent, full learning plots are shown. Full plots for all agents are located in \ref{['sec_rawresults']}.
• Figure 2: A diagram of the inverted pendulum environment.
• Figure 3: A diagram of the pendulum environment.
• Figure 4: A diagram of the three-wheel robot environment.
• Figure 5: A diagram of the two-tank environment.

Theorems & Definitions (38)

• Theorem 1
• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Remark 5
• Remark 6
• Remark 7
• Remark 8
• Theorem 2