Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations

Huy Hoang, Tien Mai, Pradeep Varakantham

TL;DR

UNIQ addresses offline learning from undesirable demonstrations by reframing imitation as maximizing a statistical distance between the learner's state-action occupancy and the undesired policy in the stationary distribution space. It adopts an offline inverse Q-learning approach, augmented with occupancy-ratio corrections and a weighted-behavior-cloning policy extraction, enabling robust learning from a small set of undesired demonstrations and a large unlabeled dataset. The method demonstrates strong safety-focused performance on Safety-Gym and MuJoCo velocity tasks, often achieving the lowest costs while maintaining competitive returns, and requires minimal hyper-parameter tuning. The work advances practical safe-RL by providing a principled, data-efficient framework for avoiding harmful trajectories in offline settings, with broader implications for healthcare and autonomous systems.

Abstract

We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines. The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ.

UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations

TL;DR

UNIQ addresses offline learning from undesirable demonstrations by reframing imitation as maximizing a statistical distance between the learner's state-action occupancy and the undesired policy in the stationary distribution space. It adopts an offline inverse Q-learning approach, augmented with occupancy-ratio corrections and a weighted-behavior-cloning policy extraction, enabling robust learning from a small set of undesired demonstrations and a large unlabeled dataset. The method demonstrates strong safety-focused performance on Safety-Gym and MuJoCo velocity tasks, often achieving the lowest costs while maintaining competitive returns, and requires minimal hyper-parameter tuning. The work advances practical safe-RL by providing a principled, data-efficient framework for avoiding harmful trajectories in offline settings, with broader implications for healthcare and autonomous systems.

Abstract

We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines. The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ.

Paper Structure

This paper contains 47 sections, 3 theorems, 41 equations, 151 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Preliminaries.
  5. MaxEnt IRL
  6. Inverse Q-learning (IQ-Learn) from expert demonstrations.
  7. UNIQ: Inverse Q-learning from Undesired Demonstrations
  8. Inverse Q-learning from Undesired Demonstrations
  9. Learning with Unlabeled Data
  10. Policy Extraction
  11. Practical Implementation
  12. Experiments
  13. Experiment setup
  14. Main comparison on Safety-Gym Tasks
  15. Mujoco Velocity benchmarks
  16. ...and 32 more sections

Key Result

Proposition 4.1

For a non-restricted feasible set of the reward function: where $d_{\psi}(\rho^\pi,\rho^{\textsc{UN}}) = \psi^*( \rho^\pi- \rho^{\textsc{UN}} )$, and $\psi^*$ is the convex conjugate of the convex function $\psi$, i.e., $\psi^*(t) = \sup_{z} \{\langle t,z\rangle - \psi(z)\}$.

Figures (151)

  • Figure 1: A general overview of offline imitation learning approaches: while prior approaches mostly aim at minimizing the distance between the learning policy and expert data (or a combination of non-expert and unlabeled data), our goal is to maximize the distance between the learning policy and an undesired policy. In UNIQ, the unlabeled dataset ($\rho^{\textsc{Mix}}$) is used in the empirical approximation of the main objective function.
  • Figure : Return
  • Figure : Return
  • Figure : Point-Goal
  • Figure : Cheetah
  • ...and 146 more figures

Theorems & Definitions (6)

  • Proposition 4.1
  • Proposition 4.2
  • Proposition 4.3
  • proof
  • proof
  • proof