Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learning

TL;DR

This work tackles strictly batch imitation learning, where no environment interaction or supplementary data are available and reward models are not used. It introduces Markov Balance-based Imitation Learning (MBIL), which leverages the Markov Balance Equation and conditional normalizing flows to estimate conditional transition densities, combining a BC-based policy loss with a dynamics-based regularizer. The resulting objective jointly enforces dynamics consistency and behavioral similarity, enabling effective learning from limited offline demonstrations and outperforming several offline IL baselines on Classic Control and MuJoCo tasks. By avoiding reward estimation and stationary distribution matching, MBIL mitigates termination and reward bias and shows strong practical potential for data-scarce, safety-critical robotics applications.

Abstract

Imitation learning (IL) is notably effective for robotic tasks where directly programming behaviors or defining optimal control costs is challenging. In this work, we address a scenario where the imitator relies solely on observed behavior and cannot make environmental interactions during learning. It does not have additional supplementary datasets beyond the expert's dataset nor any information about the transition dynamics. Unlike state-of-the-art (SOTA) IL methods, this approach tackles the limitations of conventional IL by operating in a more constrained and realistic setting. Our method uses the Markov balance equation and introduces a novel conditional density estimation-based imitation learning framework. It employs conditional normalizing flows for transition dynamics estimation and aims at satisfying a balance equation for the environment. Through a series of numerical experiments on Classic Control and MuJoCo environments, we demonstrate consistently superior empirical performance compared to many SOTA IL algorithms.

Figures (5)

• Figure 1: A high-level description of Conditional Normalizing Flow, where $CB_{i}$ is the $i^{th}$ coupling block.
• Figure 2: A high-level description of coupling block $CB_{i}$ in conditional density estimation setup.
• Figure 3: Average rewards achieved by benchmark IRL/IL/AIL and MBIL policies during real-time deployment plotted against the number of trajectories included in demonstration dataset $D$ (higher values indicate better performance).
• Figure 4: Average rewards achieved by benchmark and MBIL policies against the number of gradient updates on MuJoCo tasks with $1$ expert trajectory (higher values indicate better performance).
• Figure 5: Ablation study: Average rewards achieved by policies trained using different combinations of $\alpha$ and $\beta$ values in the MBIL objective against the number of gradient updates on MuJoCo tasks with $1$ expert trajectory on Walker2d environment (higher values indicate better performance).