Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Generalisable Imitation Learning Through Conditioned Transition Estimation and Online Behaviour Alignment

Nathan Gavenski, Matteo Leonetti, Odinaldo Rodrigues

TL;DR

The paper tackles ILfO limitations by introducing UfO, a fully unsupervised two-stage framework that learns from state transitions alone. In the reconstruction stage, a policy and a conditioned generator jointly approximate the environment dynamics to align agent transitions with teacher transitions without action labels; this is followed by an online adversarial refinement stage that uses a recurrent discriminator to further align the agent with the teacher through safe online interaction. UfO demonstrates state-of-the-art performance and the lowest trajectory variance across five MuJoCo environments, including cases where the agent surpasses the teacher, highlighting strong generalisation and robustness to unseen states. By reframing imitation through transition-conditioned learning rather than action inference, UfO mitigates behaviour-seeking biases and reduces dependency on action-level supervision, offering a general framework for observational learning with potential implications for offline RL and sim-to-real transfer.

Abstract

State-of-the-art imitation learning from observation methods (ILfO) have recently made significant progress, but they still have some limitations: they need action-based supervised optimisation, assume that states have a single optimal action, and tend to apply teacher actions without full consideration of the actual environment state. While the truth may be out there in observed trajectories, existing methods struggle to extract it without supervision. In this work, we propose Unsupervised Imitation Learning from Observation (UfO) that addresses all of these limitations. UfO learns a policy through a two-stage process, in which the agent first obtains an approximation of the teacher's true actions in the observed state transitions, and then refines the learned policy further by adjusting agent trajectories to closely align them with the teacher's. Experiments we conducted in five widely used environments show that UfO not only outperforms the teacher and all other ILfO methods but also displays the smallest standard deviation. This reduction in standard deviation indicates better generalisation in unseen scenarios.

Towards Generalisable Imitation Learning Through Conditioned Transition Estimation and Online Behaviour Alignment

TL;DR

The paper tackles ILfO limitations by introducing UfO, a fully unsupervised two-stage framework that learns from state transitions alone. In the reconstruction stage, a policy and a conditioned generator jointly approximate the environment dynamics to align agent transitions with teacher transitions without action labels; this is followed by an online adversarial refinement stage that uses a recurrent discriminator to further align the agent with the teacher through safe online interaction. UfO demonstrates state-of-the-art performance and the lowest trajectory variance across five MuJoCo environments, including cases where the agent surpasses the teacher, highlighting strong generalisation and robustness to unseen states. By reframing imitation through transition-conditioned learning rather than action inference, UfO mitigates behaviour-seeking biases and reduces dependency on action-level supervision, offering a general framework for observational learning with potential implications for offline RL and sim-to-real transfer.

Abstract

State-of-the-art imitation learning from observation methods (ILfO) have recently made significant progress, but they still have some limitations: they need action-based supervised optimisation, assume that states have a single optimal action, and tend to apply teacher actions without full consideration of the actual environment state. While the truth may be out there in observed trajectories, existing methods struggle to extract it without supervision. In this work, we propose Unsupervised Imitation Learning from Observation (UfO) that addresses all of these limitations. UfO learns a policy through a two-stage process, in which the agent first obtains an approximation of the teacher's true actions in the observed state transitions, and then refines the learned policy further by adjusting agent trajectories to closely align them with the teacher's. Experiments we conducted in five widely used environments show that UfO not only outperforms the teacher and all other ILfO methods but also displays the smallest standard deviation. This reduction in standard deviation indicates better generalisation in unseen scenarios.
Paper Structure (34 sections, 6 theorems, 8 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 34 sections, 6 theorems, 8 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Behavioural Cloning Methods
  4. Adversarial Imitation Learning Methods
  5. Unsupervised IL from Observation
  6. Reconstruction Stage
  7. Adversarial Stage
  8. Experimental Results
  9. Implementation and Metrics
  10. Results
  11. Theoretical Underpinning for Convergence
  12. UfO Setup
  13. Reconstruction Stage
  14. Adversarial Stage
  15. Discussion
  16. ...and 19 more sections

Key Result

lemma 1

Let ${\pi_\theta}$ be the policy model, $\mathcal{G}_\phi$ be the generative model, and ${\mathcal{L}}$ be the squared loss, which interleaves every $k$ iteration the parameter updates as follows: We assume that ${\pi_\theta}$ and $\mathcal{G}_\phi$ are continuously differentiable with Lipschitz-continuous gradients, and that learning rates are bounded so that parameter updates remain within comp

Figures (3)

  • Figure 1: Two-stage training cycle. (a) Each model is iteratively frozen and trained using samples from the teacher $\mathcal{T}_{\pi_\psi}$ and the agent, respectively. (b) $\mathcal{G}_\phi$ is frozen, and ${\pi_\theta}$ and $\mathcal{D}_\omega$ are trained with agent samples from the environment and the teacher's.
  • Figure 2: Training information for UfO in Hopper-v$4$.
  • Figure 3: UfO's CV and $\mathbf{AER}$ for the HalfCheetah-v$4$.

Theorems & Definitions (6)

  • lemma 1
  • lemma 2
  • lemma 3
  • lemma 4
  • lemma 5
  • lemma 6