TubeDAgger: Reducing the Number of Expert Interventions with Stochastic Reach-Tubes

TL;DR

TubeDAgger addresses the practical burden of expert interventions in interactive imitation learning by replacing the doubt classifier with precomputed stochastic reach-tubes derived from expert trajectories. The method gates autonomous control using a tube-based safety criterion, defined by a center, radius, and time-indexed ellipsoids, ensuring safety with probabilistic guarantees while avoiding environment-specific threshold tuning. Empirical results on 2D navigation and MuJoCo tasks show reduced context switches and maintained performance compared to LazyDAgger and ensembles, demonstrating robustness to threshold choices. The approach offers a principled, scalable safety mechanism for IL in high-dimensional, stochastic environments and highlights the trade-offs between tube conservativeness and learning progress. Future work includes improving temporal alignment and tube construction efficiency for larger systems.

Abstract

Interactive Imitation Learning deals with training a novice policy from expert demonstrations in an online fashion. The established DAgger algorithm trains a robust novice policy by alternating between interacting with the environment and retraining of the network. Many variants thereof exist, that differ in the method of discerning whether to allow the novice to act or return control to the expert. We propose the use of stochastic reachtubes - common in verification of dynamical systems - as a novel method for estimating the necessity of expert intervention. Our approach does not require fine-tuning of decision thresholds per environment and effectively reduces the number of expert interventions, especially when compared with related approaches that make use of a doubt classification model.

Figures (11)

• Figure 1: A schematic of the TubeDAgger approach which encompasses initial collection of expert trajectories, construction of a reachtube, and employing it as a decision boundary in place of the doubt model of LazyDAgger.
• Figure 2: Reachtube for a 2D navigation toy example. The agent start on the right has to reach the goal position on the left while avoiding the gray walls. Left: The tube is depicted in purple; yellow and green respectively show the $0.7$ and $0.2$ boundaries used by TubeDAgger. Right: Reward curves for the imitator evaluation reward (blue) and the reward achieved by the combined imitator-expert agent (orange).
• Figure 3: Boxplots showing the validation rewards for 5 runs each of LazyDAgger and TubeDAgger with different lower and upper thresholds for the action distance. Top row shows results for LazyDAgger and bottom row for TubeDAgger. When comparing to the LazyDAgger results above, we can see that TubeDAgger is more robust to the choice of threshold.
• Figure 4: Boxplots showing the percentage of novice actions at the end of training for 5 runs each with different lower and upper thresholds. The top row shows LazyDAgger and the bottom row TubeDAgger results. Again, we can see improved robustness to hyperparameter choice when compared to LazyDAgger.
• Figure 5: Reach-tubes generated by GoTube. The plots show the first two dimensions of the respective system as x- and y-axes with the z-axis denoting time.