DART: Noise Injection for Robust Imitation Learning

TL;DR

DART tackles covariate shift in imitation learning by injecting optimized noise into the supervisor’s policy, producing corrective demonstrations that resemble the robot’s eventual errors without the drawbacks of retraining after every iteration. The approach yields an off-policy learning algorithm that rivals on-policy methods in MuJoCo tasks while dramatically reducing computation and maintaining supervisor safety. Theoretical analysis shows DART reduces a bound on distributional shift, and empirical results demonstrate substantial improvements over Behavior Cloning in real grasping tasks and competitive performance with DAgger in simulation. Overall, DART offers a data-efficient, supervisor-friendly path to robust imitation learned policies for high-dimensional robotic control.

Abstract

One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this "off-policy" approach is that the robot's errors compound when drifting away from the supervisor's demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose an off-policy approach that injects noise into the supervisor's policy while demonstrating. This forces the supervisor to demonstrate how to recover from errors. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot's trained policy during data collection. We compare DART with DAgger and Behavior Cloning in two domains: in simulation with an algorithmic supervisor on the MuJoCo tasks (Walker, Humanoid, Hopper, Half-Cheetah) and in physical experiments with human supervisors training a Toyota HSR robot to perform grasping in clutter. For high dimensional tasks like Humanoid, DART can be up to $3x$ faster in computation time and only decreases the supervisor's cumulative reward by $5\%$ during training, whereas DAgger executes policies that have $80\%$ less cumulative reward than the supervisor. On the grasping in clutter task, DART obtains on average a $62\%$ performance increase over Behavior Cloning.

Figures (6)

• Figure 1: Robot Learning to reach a goal state $\mathbf{x}_G$. The grey denotes the distribution over trajectories. Left: Off-Policy learning in which the supervisor, the orange arrows, provides demonstrations. The robot, the teal arrows, deviates from the distributions and incurs high error. Middle: On-Policy which samples from the current robot's policy, the light teal arrows, to receive corrective examples from the supervisor. Right: DART, which injects noise to widen the supervisor's distribution and provides corrective examples. DART is off-policy but robust.
• Figure 2: Top: The four different locomotive domains in MuJoCo we evaluated DART on: Walker, Hopper, Half-Cheetah and Humanoid. Top Middle: The time, in seconds, to achieve the performance level reported below. DART achieves similar performance to DAgger in all 4 domains, but requires significantly less computation because it doesn't require retraining the current robot policy after each demonstration. DAgger-B reduces the computation required by less frequently training the robot, but suffers significantly in performance in domains like Humanoid. Bottom Middle: The learning curve with respect to reward obtained during training with each algorithm plotted across the number of demonstrations. Bottom: The reward obtained during collection of demonstrations for learning. DART receives near the supervisor's reward at all iterations whereas DAgger can be substantially worse in the beginning.
• Figure 3: Left: Experimental setup for the grasping in clutter task. A Toyota HSR robot uses a head-mounted RGBD camera and its arm to push obstacle objects out of the way to reach the goal object, a mustard bottle. The robot's policy for pushing objects away uses a CNN trained on images taken from the robot's Primesense camera, an example image from the robot's view point is shown in the orange box. Right: the Success Rate for Behavior Cloning, DART($\alpha=3$) and DART($\alpha =6$). DART($\alpha=3$) achieves the largest success rate.
• Figure 4: The covariate shift on the four different locomotive domains in MuJoCo we evaluated DART on: Walker, Hopper, Half-Cheetah and Humanoid. The dashed lines correspond to the surrogate losses on the supervisor distributions, $E_{p(\xi |\pi_{\theta^*},\psi^\alpha_k)} J(\theta^R,\theta^*|\xi)$, and the solid lines correspond to the loss on the robot distributions $E_{p(\xi |\pi_{\theta^R})} J(\theta^R,\theta^*|\xi)$. DART is able to reduce the covariate shift in all four domains, which is illustrated by the dash line converging to the solid line. Additionally, DAgger is also able to reduce the shift. However, Behavior Cloning is less effective at doing so, which is illustrated by a slower convergence rate.
• Figure 5: The losses of learners trained from supervisors with randomly sampled covariance matrices are compared against the losses of DART. Again, the dashed lines represent the loss on the supervisor disributions while the solid lines represent the loss on the robot distributions. When the simulated error is well-chose, such as $\text{Tr}(\Sigma) = 0.5$ in these experiments, the performance matches that of DART and the covariate shift is reduced. However higher or lower levels of the noise can cause drastically higher losses.

Theorems & Definitions (5)

• Lemma 4.1
• Proposition 4.1
• Lemma 4.1
• Lemma 4.2
• Proposition 4.1