Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CubeDAgger: Improved Robustness of Interactive Imitation Learning without Violation of Dynamic Stability

Taisuke Kobayashi

TL;DR

CubeDAgger addresses robustness and dynamic stability in interactive imitation learning by replacing brittle expert–agent switching with a three-component framework: controlling ensemble uncertainty, a consensus-based continuous action selection, and time-consistent exploration via autoregressive colored noise. It introduces an inequality constraint and regularization to bound ensemble deviations, an L_p-based consensus mechanism to blend multiple action candidates, and a red-noise exploration strategy that remains compatible with human demonstrations. In Mujoco simulations across Pusher, HalfCheetah, and HexaAnt, CubeDAgger demonstrates improved robustness of learned policies with acceptable retention and maintained stability during data collection, outperforming EnsembleDAgger and its ablations. These contributions advance practical, data-efficient, and safe interactive imitation learning for dynamic robotic systems.

Abstract

Interactive imitation learning makes an agent's control policy robust by stepwise supervisions from an expert. The recent algorithms mostly employ expert-agent switching systems to reduce the expert's burden by limitedly selecting the supervision timing. However, the precise selection is difficult and such a switching causes abrupt changes in actions, damaging the dynamic stability. This paper therefore proposes a novel method, so-called CubeDAgger, which improves robustness while reducing dynamic stability violations by making three improvements to a baseline method, EnsembleDAgger. The first improvement adds a regularization to explicitly activate the threshold for deciding the supervision timing. The second transforms the expert-agent switching system to an optimal consensus system of multiple action candidates. Third, autoregressive colored noise to the actions is introduced to make the stochastic exploration consistent over time. These improvements are verified by simulations, showing that the learned policies are sufficiently robust while maintaining dynamic stability during interaction.

CubeDAgger: Improved Robustness of Interactive Imitation Learning without Violation of Dynamic Stability

TL;DR

CubeDAgger addresses robustness and dynamic stability in interactive imitation learning by replacing brittle expert–agent switching with a three-component framework: controlling ensemble uncertainty, a consensus-based continuous action selection, and time-consistent exploration via autoregressive colored noise. It introduces an inequality constraint and regularization to bound ensemble deviations, an L_p-based consensus mechanism to blend multiple action candidates, and a red-noise exploration strategy that remains compatible with human demonstrations. In Mujoco simulations across Pusher, HalfCheetah, and HexaAnt, CubeDAgger demonstrates improved robustness of learned policies with acceptable retention and maintained stability during data collection, outperforming EnsembleDAgger and its ablations. These contributions advance practical, data-efficient, and safe interactive imitation learning for dynamic robotic systems.

Abstract

Interactive imitation learning makes an agent's control policy robust by stepwise supervisions from an expert. The recent algorithms mostly employ expert-agent switching systems to reduce the expert's burden by limitedly selecting the supervision timing. However, the precise selection is difficult and such a switching causes abrupt changes in actions, damaging the dynamic stability. This paper therefore proposes a novel method, so-called CubeDAgger, which improves robustness while reducing dynamic stability violations by making three improvements to a baseline method, EnsembleDAgger. The first improvement adds a regularization to explicitly activate the threshold for deciding the supervision timing. The second transforms the expert-agent switching system to an optimal consensus system of multiple action candidates. Third, autoregressive colored noise to the actions is introduced to make the stochastic exploration consistent over time. These improvements are verified by simulations, showing that the learned policies are sufficiently robust while maintaining dynamic stability during interaction.
Paper Structure (25 sections, 14 equations, 4 figures, 1 algorithm)

This paper contains 25 sections, 14 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. DAgger family
  4. DART family
  5. Integration with reinforcement learning
  6. Preliminaries
  7. Behavioral cloning
  8. Interactive imitation learning
  9. EnsembleDAgger
  10. Proposed method
  11. Overview
  12. Controlled ensemble uncertainty
  13. Formulation
  14. Optimization for satisfying inequality constraint
  15. Consensus system among ensemble actions
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Geometries for designing $p_i$ and $w_k$
  • Figure 2: Time consistency of red noise ($\Delta t=0.05$ and $T=3$)
  • Figure 3: Trajectories during data collection (dashed lines: the experts' average scores)
  • Figure 4: Statistical evaluation with 21 random seeds for each condition (dashed lines: the experts' worst-case scores)