Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual RL: Unification and New Methods for Reinforcement and Imitation Learning

Harshit Sikchi, Qinqing Zheng, Amy Zhang, Scott Niekum

TL;DR

<3-5 sentence high-level summary> Dual RL provides a unified duality-based lens to view both offline reinforcement learning and offline imitation learning, casting policy performance as an unconstrained visitation-distribution optimization. The authors show that many existing algorithms are instances of dual-Q or dual-V with different divergences and update rules, enabling principled use of arbitrary off-policy data. They identify a discriminatory coverage limitation in IL and propose ReCOIL, a discriminator-free method, along with f-DVL, a family of stable implicit maximizers for offline RL. Extensive experiments on MuJoCo locomotion and manipulation tasks validate improved performance and training stability, highlighting the practical impact of the Dual-RL perspective for robust, data-efficient learning.

Abstract

The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.

Dual RL: Unification and New Methods for Reinforcement and Imitation Learning

TL;DR

<3-5 sentence high-level summary> Dual RL provides a unified duality-based lens to view both offline reinforcement learning and offline imitation learning, casting policy performance as an unconstrained visitation-distribution optimization. The authors show that many existing algorithms are instances of dual-Q or dual-V with different divergences and update rules, enabling principled use of arbitrary off-policy data. They identify a discriminatory coverage limitation in IL and propose ReCOIL, a discriminator-free method, along with f-DVL, a family of stable implicit maximizers for offline RL. Extensive experiments on MuJoCo locomotion and manipulation tasks validate improved performance and training stability, highlighting the practical impact of the Dual-RL perspective for robust, data-efficient learning.

Abstract

The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.
Paper Structure (76 sections, 11 theorems, 119 equations, 21 figures, 10 tables, 3 algorithms)

This paper contains 76 sections, 11 theorems, 119 equations, 21 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Off-Policy Methods for IL
  4. Off-Policy Methods for RL
  5. Preliminaries
  6. Value Functions and Bellman Operators
  7. Reinforcement Learning via Lagrangian Duality
  8. A unified perspective on RL and IL through Duality
  9. Dual Formulation for existing imitation learning algorithms
  10. a. Offline IL with Expert Data Only
  11. b. Offline IL with Additional Suboptimal Data
  12. Dual Formulation for existing reinforcement learning algorithms
  13. A consequence of unification in RL:
  14. ReCOIL: Imitation Learning from Arbitrary Experience
  15. A Bellman Consistent Energy-Based Model (EBM) View for ReCOIL
  16. ...and 61 more sections

Key Result

proposition 1

IQLearn garg2021iq is an instance of $\texttt{dual-Q}$ using the semi-gradient For an overview of semi-gradient vs full-gradient methods please refer to Appendix ap:semi_gradient_info. update rule with a (soft) Bellman operator, where $r(s,a)=0 \, \forall s \in \mathcal{S}, a \in \mathcal{A}$, $d^O=

Figures (21)

  • Figure 1: (a) [Left] shows an MDP that starts at the leftmost state and transitions to one of the five absorbing states on the right. Under the given expert and replay/offline visitation we study if a prespecified policy's visitation can be inferred whose ground truth visitation is known [Right] shows MSE error plots with policy's ground truth visitation where ReCOIL perfectly infers $d^\pi$ whereas a method that only relies on expert data or the replay data with the coverage assumption fails. Results averaged over 100 seeds. More details in Appendix \ref{['ap:density_ratio_estimation']} (b) Recovered $R$ and $V^*$ on a simple grid-world environment by ReCOIL.
  • Figure 2: XQL training diverges due to the numerical instability of its loss function. $f$-DVL fixes this problem by using more well-behaved $f$-divergences.
  • Figure 3: Augmenting SAC with expert data at the start of training destabilizes value function learning (r), but dual-RL approaches can make effective use of the additional data to learn performant policy (l).
  • Figure 4: We show that a number of prior methods can be understood as a special case of the dual RL framework. Based on this framework, we also propose new methods addressing the shortcomings of previous works (boxed in green).
  • Figure 5: Illustration of a family of implicit maximizers corresponding to different $f$-divergences. The underlying data distribution is a truncated Gaussian TN with mean $0$, variance $1$ and a truncation range $(-2, 2)$. We sample 10000 data points from TN and compute the solution $v_\lambda$ of Problem \ref{['eq:implicit_maximization_general']}. As $\lambda \to 1$, the solution $v_\lambda$ becomes a more accurate estimation for the supremum of the random variable $x$.
  • ...and 16 more figures

Theorems & Definitions (15)

  • proposition 1
  • proposition 2
  • theorem 1
  • proposition 3
  • proposition 4
  • proof
  • proposition 4
  • proof
  • proposition 4
  • proof
  • ...and 5 more