Inverse Q-Learning Done Right: Offline Imitation Learning in $Q^π$-Realizable MDPs

TL;DR

This work tackles offline imitation learning by dropping the common expert realizability assumption and instead leveraging a structural property: linear $Q^\pi$-realizable MDPs. It introduces SPOIL, a primal-dual saddle-point algorithm whose actor-critic updates enable efficient learning from expert data without interacting with the environment. In the linear case, SPOIL achieves an $\varepsilon$-accurate match to the expert with $\mathcal{O}(\varepsilon^{-2})$ samples, independent of the expert class, while extending to general (nonlinear) function approximation yields $\mathcal{O}(\varepsilon^{-4})$ sample complexity due to higher class capacity. Empirical results on both synthetic and continuous-state benchmarks show SPOIL is competitive with state-of-the-art offline IL methods and can surpass behavioral cloning, highlighting the practical viability of exploiting environment structure for imitation learning.

Abstract

We study the problem of offline imitation learning in Markov decision processes (MDPs), where the goal is to learn a well-performing policy given a dataset of state-action pairs generated by an expert policy. Complementing a recent line of work on this topic that assumes the expert belongs to a tractable class of known policies, we approach this problem from a new angle and leverage a different type of structural assumption about the environment. Specifically, for the class of linear $Q^π$-realizable MDPs, we introduce a new algorithm called saddle-point offline imitation learning (\SPOIL), which is guaranteed to match the performance of any expert up to an additive error $\varepsilon$ with access to $\mathcal{O}(\varepsilon^{-2})$ samples. Moreover, we extend this result to possibly non-linear $Q^π$-realizable MDPs at the cost of a worse sample complexity of order $\mathcal{O}(\varepsilon^{-4})$. Finally, our analysis suggests a new loss function for training critic networks from expert data in deep imitation learning. Empirical evaluations on standard benchmarks demonstrate that the neural net implementation of \SPOIL is superior to behavior cloning and competitive with state-of-the-art algorithms.

Figures (6)

• Figure 1: Experiments with simple and complex experts. Curves are averaged across $10$ seeds.
• Figure 2: Experiments in continuous states domains. Curves are averaged across $10$ seeds.
• Figure 3: Comparison of linear and quadratic softmax policies with $A = 5$ actions and features $\varphi(a) = a - 3$.
• Figure 4: Instance in which BC with simple expert class can outperform SPOIL
• Figure 5: Experiments in continuous states domains. Curves are averaged across $10$ seeds.

Theorems & Definitions (33)

• Lemma 1
• Definition 1: Covering number
• Proposition 1
• proof
• Theorem 1
• Theorem 2
• Lemma 2
• Lemma 3
• proof : Proof sketch of Theorem \ref{['thm:main_formal']}
• Lemma 4