Trajectory Entropy Reinforcement Learning for Predictable and Robust Control

TL;DR

This work introduces a novel inductive bias towards simple policies in reinforcement learning by minimizing the entropy of entire action trajectories, corresponding to the number of bits required to describe information in action trajectories after the agent observes state trajectories.

Abstract

Simplicity is a critical inductive bias for designing data-driven controllers, especially when robustness is important. Despite the impressive results of deep reinforcement learning in complex control tasks, it is prone to capturing intricate and spurious correlations between observations and actions, leading to failure under slight perturbations to the environment. To tackle this problem, in this work we introduce a novel inductive bias towards simple policies in reinforcement learning. The simplicity inductive bias is introduced by minimizing the entropy of entire action trajectories, corresponding to the number of bits required to describe information in action trajectories after the agent observes state trajectories. Our reinforcement learning agent, Trajectory Entropy Reinforcement Learning, is optimized to minimize the trajectory entropy while maximizing rewards. We show that the trajectory entropy can be effectively estimated by learning a variational parameterized action prediction model, and use the prediction model to construct an information-regularized reward function. Furthermore, we construct a practical algorithm that enables the joint optimization of models, including the policy and the prediction model. Experimental evaluations on several high-dimensional locomotion tasks show that our learned policies produce more cyclical and consistent action trajectories, and achieve superior performance, and robustness to noise and dynamic changes than the state-of-the-art.

Figures (8)

• Figure 1: Performance comparison, and action trajectory visualizations of our method (TERL) and its version without the simplicity inductive bias (SAC) on the Humanoid Walk task. Our method learns a policy that produces more periodic and consistent action sequences, improving performance on the Humanoid Walk task.
• Figure 2: Our TERL agent minimizes the entropy of action trajectories conditioned on trajectories of state representations. The action trajectories are determined by the policy, while the representations are generated by a state encoder.
• Figure 3: We use one objective to jointly optimize our policy, the encoder, and the lower bound of the trajectory entropy.
• Figure 4: We evaluate our method and previous methods on six high-dimensional locomotion tasks: H1 Walk, Hopper Stand, Walker Walk, Cheetah Run, Humanoid Walk, and Quadruped Walk.
• Figure 5: Zero-shot robustness to changes in body gravity (left), action noise (middle), and observation noise (right) on 6 locomotion control tasks. This plot shows the normalized mean rewards averaged over 20 independent runs and 6 tasks, with error bar representing 90% confidence interval. To make comparisons across tasks, we normalize rewards by the reward achieved by the best method on each task. For each run, we collect 30 evaluation trajectories. TERL achieves better aggregated performance than all baselines when environments are disturbed by action noise, mass changes, and small observation noise.