Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Imitation Learning in Discounted Linear MDPs without exploration assumptions

Luca Viano, Stratis Skoulakis, Volkan Cevher

TL;DR

This paper tackles imitation learning in infinite-horizon discounted Linear MDPs with linear costs and introduces ILARL, an algorithm that eliminates reliance on the persistent-excitation exploration assumption and tightens the ε-dependence from $\tilde{O}(\epsilon^{-5})$ to $\tilde{O}(\epsilon^{-4})$, by connecting imitation learning to online learning with adversarial losses. It also delivers a stronger finite-horizon bound and demonstrates superior empirical performance against standard baselines. The core technique blends online-to-batch conversion with a regret-decomposition framework, enabling two no-regret players to coordinate between updating cost parameters and updating policies, respectively. For finite horizons, a best-response extension BRIG yields a further improvement in sample complexity to $\tilde{O}(H^4 d^3 \epsilon^{-2})$, while maintaining computational practicality. Overall, the work advances theoretical guarantees and practical performance for IL in linear MDPs, with potential impacts on robotics and autonomous decision-making where reward specification is challenging.

Abstract

We present a new algorithm for imitation learning in infinite horizon linear MDPs dubbed ILARL which greatly improves the bound on the number of trajectories that the learner needs to sample from the environment. In particular, we remove exploration assumptions required in previous works and we improve the dependence on the desired accuracy $ε$ from $\mathcal{O}(ε^{-5})$ to $\mathcal{O}(ε^{-4})$. Our result relies on a connection between imitation learning and online learning in MDPs with adversarial losses. For the latter setting, we present the first result for infinite horizon linear MDP which may be of independent interest. Moreover, we are able to provide a strengthen result for the finite horizon case where we achieve $\mathcal{O}(ε^{-2})$. Numerical experiments with linear function approximation shows that ILARL outperforms other commonly used algorithms.

Imitation Learning in Discounted Linear MDPs without exploration assumptions

TL;DR

This paper tackles imitation learning in infinite-horizon discounted Linear MDPs with linear costs and introduces ILARL, an algorithm that eliminates reliance on the persistent-excitation exploration assumption and tightens the ε-dependence from to , by connecting imitation learning to online learning with adversarial losses. It also delivers a stronger finite-horizon bound and demonstrates superior empirical performance against standard baselines. The core technique blends online-to-batch conversion with a regret-decomposition framework, enabling two no-regret players to coordinate between updating cost parameters and updating policies, respectively. For finite horizons, a best-response extension BRIG yields a further improvement in sample complexity to , while maintaining computational practicality. Overall, the work advances theoretical guarantees and practical performance for IL in linear MDPs, with potential impacts on robotics and autonomous decision-making where reward specification is challenging.

Abstract

We present a new algorithm for imitation learning in infinite horizon linear MDPs dubbed ILARL which greatly improves the bound on the number of trajectories that the learner needs to sample from the environment. In particular, we remove exploration assumptions required in previous works and we improve the dependence on the desired accuracy from to . Our result relies on a connection between imitation learning and online learning in MDPs with adversarial losses. For the latter setting, we present the first result for infinite horizon linear MDP which may be of independent interest. Moreover, we are able to provide a strengthen result for the finite horizon case where we achieve . Numerical experiments with linear function approximation shows that ILARL outperforms other commonly used algorithms.
Paper Structure (40 sections, 19 theorems, 120 equations, 4 figures, 1 table, 6 algorithms)

This paper contains 40 sections, 19 theorems, 120 equations, 4 figures, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Assumption. Persistent excitation
  3. Our contribution
  4. Related Works
  5. Background and Notation
  6. Setting
  7. Main Results and techniques
  8. Technique overview
  9. Online-to-batch conversion
  10. Regret decomposition
  11. Imitation Learning via no-regret algorithms.
  12. Improved algorithm for finite horizon
  13. Warm up: Online Learning in Adversarial Linear MDP
  14. Analysis
  15. Extension to the Infinite Horizon Setting.
  16. ...and 25 more sections

Key Result

Theorem 1

Under Assumptions ass:LinMDP,ass:cost, there exists an algorithm, i.e. alg:infinite_imitation, such that after using $\widetilde{\mathcal{O}}\left({\frac{\log \left| {\mathcal{A}} \right| d^3}{(1 - \gamma)^8 \epsilon^4}}\right)$ state action pairs from the MDP and using $\widetilde{\mathcal{O}}\left

Figures (4)

  • Figure 1: Experiments on a continuous gridworld with a stochastic expert.The $y$-axis reports the normalized return. $1$ correpsonds to the expert performance and $0$ to the uniform policy one.
  • Figure 2: Experiments on the continuous gridworld with one trajectory from a deterministic expert.
  • Figure 3: Graphical representation of $- \mathbf{c_{\textup{true}}}$ of the linear MDP used in \ref{['fig:linMDP', 'fig:noisy_lin_MDP']}.
  • Figure 4: Experiment in finite horizon setting to assess the better efficiency of BRIG.

Theorems & Definitions (42)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • Remark 1
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Theorem 5
  • Remark 2
  • ...and 32 more