Efficient Off-Policy Learning for High-Dimensional Action Spaces

TL;DR

This work tackles data inefficiency in off-policy reinforcement learning with high-dimensional action spaces by proposing Vlearn, a method that uses only a state-value function as the critic. It introduces an upper-bound, importance-weighted loss (L_WIS) for learning the V-function from off-policy data and couples it with a robust policy-update framework that includes twin critics, delayed updates, and trust-region constraints. The approach yields improved sample efficiency, stability, and final performance across challenging high-dimensional benchmarks, outperforming Q-function–based baselines and V-trace in several tasks. While effective, the method remains data-hungry and the authors discuss future directions such as offline RL extensions, ensembles, and distributional variants to further enhance stability and applicability.

Abstract

Existing off-policy reinforcement learning algorithms often rely on an explicit state-action-value function representation, which can be problematic in high-dimensional action spaces due to the curse of dimensionality. This reliance results in data inefficiency as maintaining a state-action-value function in such spaces is challenging. We present an efficient approach that utilizes only a state-value function as the critic for off-policy deep reinforcement learning. This approach, which we refer to as Vlearn, effectively circumvents the limitations of existing methods by eliminating the necessity for an explicit state-action-value function. To this end, we leverage a weighted importance sampling loss for learning deep value functions from off-policy data. While this is common for linear methods, it has not been combined with deep value function networks. This transfer to deep methods is not straightforward and requires novel design choices such as robust policy updates, twin value function networks to avoid an optimization bias, and importance weight clipping. We also present a novel analysis of the variance of our estimate compared to commonly used importance sampling estimators such as V-trace. Our approach improves sample complexity as well as final performance and ensures consistent and robust performance across various benchmark tasks. Eliminating the state-action-value function in Vlearn facilitates a streamlined learning process, yielding high-return agents.

Figures (6)

• Figure 1: To provide intuition on the differences between Vlearn and V-trace, we consider the following example for different importance ratios $\rho$. Suppose for a state $s_t$ the Bellman target is $r(s,a) + \gamma V_{\bar{\theta}}(s_{t+1}) = 4$, the target critic predicts $V_{\bar{\theta}}(s_t) = -6$ and we plot the loss for values of $V_{\theta}(s)$ for Vlearn and V-trace. For on-policy samples ($\rho=1.0$), both losses are the same. However, for increasingly off-policy samples ($\rho \to 0)$, we see how V-trace increasingly relies on the target critic, shifting the optimal value towards it. Vlearn, on the other hand, simply reduces the scale of the loss and thus the importance of the sample, making Vlearn more robust to errors in the target critic.
• Figure 2: Mean over $10$ seeds and 95% bootstrapped confidence intervals for the high-dimensional Gymnasium tasks, the $38$-dimensional dog tasks. Vlearn consistently achieves a better asymptotic performance for all tasks. For the dog tasks and even struggle to learn a consistent policy. Compared to V-trace our method is significantly more stable and achieves a better final performance.
• Figure 3: Performance on the $39$-dimensional MyoHand from MyoSuite. Left. and 95% bootstrapped confidence intervals for the aggregated performance over all $10$ MyoHand tasks. Right. Mean over $10$ seeds and 95% bootstrapped confidence intervals for the individual performances of all MyoHand tasks. While Vlearn does not outperform all baselines on all tasks, it performs well across all tasks and achieves the highest aggregated performance.
• Figure 4: Left. Ablation study on the impact of replay buffer size on policy performance. For smaller replay buffers learning becomes unstable or does not converge, while larger sizes tend to lead to similar final performances. Right. Investigation in the importance of various design choices of Vlearn on Ant-v4 and Humanoid-v4.
• Figure 5: Mean over $10$ seeds and 95% bootstrapped confidence intervals for the low-dimensional Gymnasium tasks. While achieves the best performance on the HalfCheetah-v4, Vlearn still performs similar to and achieves and equivalent performance for Hopper-v4 and Walker2D-v4. V-trace once again cannot make any meaningful progress in the off-policy setting.

Theorems & Definitions (1)

• Theorem 1