Maximum Entropy RL (Provably) Solves Some Robust RL Problems

TL;DR

This work formalizes robustness in reinforcement learning by showing that maximum entropy RL (MaxEnt RL) inherently bounds a class of robust objectives under both reward and dynamics perturbations. By introducing pessimistic reward formulations and entropy-based considerations, the authors derive theoretical lower bounds and robust sets, and they validate these findings with numerical experiments that compare favorably to specialized robust RL methods. The results illuminate why stochastic policies from MaxEnt RL can be more robust in disturbed environments, and they provide guidance on how entropy regularization parameters influence robustness. While not universally superior, MaxEnt RL offers a simple, principled approach to robust RL with formal guarantees and practical evidence of effectiveness.

Abstract

Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

Figures (12)

• Figure 1: MaxEnt RL is robust to disturbances.(Left) We applied both standard RL and MaxEnt RL to a manipulation task without obstacles, but added obstacles (red squares) during evaluation. We then plot the position of the object when evaluating the learned policies (Center) on the original environment and (Right) on the new environment with an obstacle. The stochastic policy learned by MaxEnt RL often navigates around the obstacle, whereas the deterministic policy from standard RL almost always collides with the obstacle.
• Figure 2: MaxEnt RL and Robustness to Adversarial Reward Functions: (Left) Applying MaxEnt RL to one reward function (red dot) yields a policy that is guaranteed to get high reward on many other reward functions (blue curve). (Center) For each reward function $(r(a=1), r(a=2))$ on that blue curve, we evaluate the expected return of a stochastic policy. The robust RL problem (for rewards) is to choose the policy whose worst-case reward (dark blue line) is largest. (Right) Plotting the MaxEnt RL objective (Eq. \ref{['eq:maxent-obj']}) for those same policies, we observe that the MaxEnt RL objective is identical to the robust RL objective .
• Figure 3: MaxEnt RL is competitive with prior robust RL methods.
• Figure 4: Robustness to changes in the dynamics: MaxEnt RL policies learn many ways of solving a task, making them robust to perturbations such as (Left) new obstacles and (Right) changes in the goal location.
• Figure 5: MaxEnt RL is not standard RL + noise.

Theorems & Definitions (14)

• Theorem 4.1
• Theorem 4.2
• Corollary 4.2.1
• Lemma 4.3
• Lemma A.1
• proof
• Lemma A.2
• proof
• proof
• proof