Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agnostic System Identification for Model-Based Reinforcement Learning

Stephane Ross, J. Andrew Bagnell

TL;DR

This work tackles agnostic model-based reinforcement learning by reducing it to a no-regret online learning problem through an iterative data-collection scheme (DAgger-style) that uses a balanced exploration distribution. It provides theoretical bounds linking policy performance to training losses and distribution mismatch, showing that no-regret online learners can achieve near-optimal control as data grows. The approach yields bounds that scale with model class complexity rather than MDP size and demonstrates practical efficacy on a simulated helicopter task, outperforming batch methods and mitigating train-test mismatch. The result is a simple, scalable, and principled MBRL algorithm with strong agnostic guarantees and broad applicability.

Abstract

A fundamental problem in control is to learn a model of a system from observations that is useful for controller synthesis. To provide good performance guarantees, existing methods must assume that the real system is in the class of models considered during learning. We present an iterative method with strong guarantees even in the agnostic case where the system is not in the class. In particular, we show that any no-regret online learning algorithm can be used to obtain a near-optimal policy, provided some model achieves low training error and access to a good exploration distribution. Our approach applies to both discrete and continuous domains. We demonstrate its efficacy and scalability on a challenging helicopter domain from the literature.

Agnostic System Identification for Model-Based Reinforcement Learning

TL;DR

This work tackles agnostic model-based reinforcement learning by reducing it to a no-regret online learning problem through an iterative data-collection scheme (DAgger-style) that uses a balanced exploration distribution. It provides theoretical bounds linking policy performance to training losses and distribution mismatch, showing that no-regret online learners can achieve near-optimal control as data grows. The approach yields bounds that scale with model class complexity rather than MDP size and demonstrates practical efficacy on a simulated helicopter task, outperforming batch methods and mitigating train-test mismatch. The result is a simple, scalable, and principled MBRL algorithm with strong agnostic guarantees and broad applicability.

Abstract

A fundamental problem in control is to learn a model of a system from observations that is useful for controller synthesis. To provide good performance guarantees, existing methods must assume that the real system is in the class of models considered during learning. We present an iterative method with strong guarantees even in the agnostic case where the system is not in the class. In particular, we show that any no-regret online learning algorithm can be used to obtain a near-optimal policy, provided some model achieves low training error and access to a good exploration distribution. Our approach applies to both discrete and continuous domains. We demonstrate its efficacy and scalability on a challenging helicopter domain from the literature.

Paper Structure

This paper contains 9 sections, 6 theorems, 7 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. A Simple Batch Algorithm
  4. Analysis
  5. No-Regret Methods for Agnostic MBRL
  6. Analysis
  7. Discussion
  8. Experiments on Helicopter Domain
  9. Conclusion

Key Result

Lemma 3.1

$\epsilon^{\textrm{L1}}_{\textrm{prd}} \leq \sqrt{2\epsilon^{\textrm{KL}}_{\textrm{prd}}}$ and $\epsilon^{\textrm{L1}}_{\textrm{prd}} \leq 2\epsilon^{\textrm{cls}}_{\textrm{prd}}$. The latter holds with equality if $\ell$ is the 0-1 loss.

Figures (2)

  • Figure 1: Example train-test mismatch in a helicopter domain. Train: model is fit based on samples near the desired trajectory, e.g. from watching an expert. Test: learned policy ends up in new regions where model is bad, leading to poor control performance.
  • Figure 2: Average total cost on test trajectories as a function of data collected so far, averaged over 20 repetitions of the experiments, each starting with a different random seed (all approaches use the same 20 seeds) From top to bottom: hover with no delay, hover with delay of 1, nose-in funnel. $D_t$, $D_e$ and $D_{en}$ denotes DAgger using exploration distribution $\nu_t$, $\nu_e$ and $\nu_{en}$ respectively, similarly $B_t$, $B_e$ and $B_{en}$ for the Batch algorithm, $A$ for Abbeel's algorithm, and $L$ for the linearized model's optimal controller.

Theorems & Definitions (6)

  • Lemma 3.1
  • Theorem 3.1
  • Corollary 3.1
  • Lemma 4.1
  • Theorem 4.1
  • Lemma 4.2