vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvement

TL;DR

This work addresses gradient-direction disagreements that arise when using ensemble critics in policy-improvement. It introduces a directional-uncertainty metric by modeling per-transition gradient directions with the von Mises-Fisher distribution and proposes vMFER, a resampling-based method that favors low-uncertainty transitions. The approach is a plug-in enhancement for actor-critic algorithms such as TD3 and SAC and demonstrates consistent improvements on Mujoco tasks, including when combined with PER and HER in sparse-reward settings. Overall, vMFER provides a practical, scalable way to improve learning efficiency by leveraging directional information from ensemble critics.

Abstract

Reinforcement Learning (RL) is a widely employed technique in decision-making problems, encompassing two fundamental operations -- policy evaluation and policy improvement. Enhancing learning efficiency remains a key challenge in RL, with many efforts focused on using ensemble critics to boost policy evaluation efficiency. However, when using multiple critics, the actor in the policy improvement process can obtain different gradients. Previous studies have combined these gradients without considering their disagreements. Therefore, optimizing the policy improvement process is crucial to enhance learning efficiency. This study focuses on investigating the impact of gradient disagreements caused by ensemble critics on policy improvement. We introduce the concept of uncertainty of gradient directions as a means to measure the disagreement among gradients utilized in the policy improvement process. Through measuring the disagreement among gradients, we find that transitions with lower uncertainty of gradient directions are more reliable in the policy improvement process. Building on this analysis, we propose a method called von Mises-Fisher Experience Resampling (vMFER), which optimizes the policy improvement process by resampling transitions and assigning higher confidence to transitions with lower uncertainty of gradient directions. Our experiments demonstrate that vMFER significantly outperforms the benchmark and is particularly well-suited for ensemble structures in RL.

Figures (9)

• Figure 1: Multiple Q-values and their corresponding gradients $\frac{\partial Q_i(s_t,a)}{\partial a}$ are generated by the ensemble critic function for a given state input $s_t$, with each Q-value and gradient pair corresponding to a different action $a$. Left: The ensemble critic function $Q_1(s_t,\cdot)$ and $Q_2(s_t,\cdot)$, which are depicted as multi-dimensional surfaces. Right Upper:The gradients of the ensemble critics, represented as arrows with varied colors on the contour plots of the Q-values, illustrate the direction and magnitude of the action-value function's sensitivity to changes in action space. Right Lower: The angles between the gradients, $\frac{\partial Q_1(s_t,a)}{\partial a}$ and $\frac{\partial Q_2(s_t,a)}{\partial a}$, are used to quantify the uncertainty of gradients for different actions $a$.
• Figure 2: A toy experiment illustrating the advantage of considering the uncertainty of gradient directions on the learning efficiency of policy improvement. The experiment compares three approaches: 'Uniform' involves uniformly sampling from the transitions, 'Uncertainty' utilizes only transitions with low uncertainty of gradient directions, and 'Oracle' employs only transitions that update the action in the direction of the optimal action.
• Figure 3: Examining the performance of vMFER on the Mujoco environment. The baseline curves represents pure TD3 or SAC, while vMFER (uncertainty) and vMFER (rank) represent two distinct forms of vMFER utilized in policy improvement combined with baseline.
• Figure 4: An experiment conducted in the Mujoco environment to explore the effect of various forms of VMFER on policy improvement. The experiment builds upon the PER combined with SAC.
• Figure 5: The impact of ensemble number on vMFER.