When does predictive inverse dynamics outperform behavior cloning?

TL;DR

This work addresses offline imitation learning, where BC struggles with limited expert data. It analyzes predictive inverse dynamics models (PIDM), which decompose decision making into a future-state predictor and an inverse dynamics policy, and shows that conditioning actions on predicted future states reduces variance at the cost of potential bias from imperfect predictions. The authors derive a bias-variance framework with a key EPE gap $\Delta$ and a sample-efficiency bound $\eta$, proving that PIDM can be at least as sample-efficient as BC under controllable predictor bias and often outperforms BC when additional data sources reduce bias. Empirically, PIDM achieves substantial sample-efficiency gains in both 2D navigation tasks and a high-dimensional 3D video game, confirming the theory and illustrating practical benefits for offline imitation with limited demonstrations.

Abstract

Behavior cloning (BC) is a practical offline imitation learning method, but it often fails when expert demonstrations are limited. Recent works have introduced a class of architectures named predictive inverse dynamics models (PIDM) that combine a future state predictor with an inverse dynamics model (IDM). While PIDM often outperforms BC, the reasons behind its benefits remain unclear. In this paper, we provide a theoretical explanation: PIDM introduces a bias-variance tradeoff. While predicting the future state introduces bias, conditioning the IDM on the prediction can significantly reduce variance. We establish conditions on the state predictor bias for PIDM to achieve lower prediction error and higher sample efficiency than BC, with the gap widening when additional data sources are available. We validate the theoretical insights empirically in 2D navigation tasks, where BC requires up to five times (three times on average) more demonstrations than PIDM to reach comparable performance; and in a complex 3D environment in a modern video game with high-dimensional visual inputs and stochastic transitions, where BC requires over 66\% more samples than PIDM.

Figures (14)

• Figure 1: (a) Visualization of selected milestones from the "Tour" task in a 3D video game with stochastic transitions and real-time inference. (b) Sample efficiency curves (mean $\pm$ std) for PIDM and BC, with BC needing 66% more samples to achieve 80% success rate.
• Figure 2: (a) BC learns a policy conditioned on the current state. (b) IDM learns a policy conditioned on the current and future state $k$ steps ahead. (c) Forward models predict a future state (representation) given a state and action. Note (b) IDM and (c) forward models can serve as auxiliary objectives to learn an encoder that provides effective state representations. (d) PIDM represents an alternative to BC consisting of a state predictor, akin to an action-free forward model, that predicts future state representations, and an IDM policy.
• Figure 3: Visualization of 2D navigation environment. (a) Tasks require the agent (blue box) to navigate to reach the goals (red boxes) in a particular order. (b) - (e) Visualizations of all four tasks and the traces of the 50 trajectories within the datasets.
• Figure 4: Performance per number of training demonstrations for BC and PIDM in four tasks trained on human datasets. Lines and shading correspond to the average and standard deviation across 20 seeds. We further visualize the number of samples required by PIDM and BC to reach 90% of the highest achievable performance with vertical dotted lines.
• Figure 5: State-wise EPE gaps $\Delta(s)$ (\ref{['eq:delta_per_state']}) for Four room and Multiroom datasets. We observe large gaps in states surrounding the goals where human actions are more diverse.

Theorems & Definitions (12)

• Theorem 1
• Corollary 1
• Theorem 2
• Theorem 3
• Corollary 2
• Theorem 1
• Corollary 1
• Lemma 1
• Theorem 2
• Lemma 2