Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrations

TL;DR

Counterfactual Behavior Cloning (Counter-BC) tackles offline imitation learning from imperfect human demonstrations by expanding each demonstrated action into a counterfactual set within a radius $\Delta$ and using a classifier grounded in the restricted counterfactual policy $\hat{\pi}_\theta$. The method derives a loss that minimizes entropy over counterfactual actions while keeping the full policy near the counterfactual explanations, enabling recovery of the human’s underlying intended policy $\pi^*$. The authors provide theoretical justification, show that Counter-BC generalizes prior BC-based approaches, and validate the approach across simulations with synthetic and real human data, plus a real-world air hockey user study. Results indicate Counter-BC yields more proficient, robust policies when learning from crowds or varying teacher skill, with a tunable trade-off controlled by $\Delta$ that balances fidelity to demonstrations against simplicity of explanation.

Abstract

Learning from humans is challenging because people are imperfect teachers. When everyday humans show the robot a new task they want it to perform, humans inevitably make errors (e.g., inputting noisy actions) and provide suboptimal examples (e.g., overshooting the goal). Existing methods learn by mimicking the exact behaviors the human teacher provides -- but this approach is fundamentally limited because the demonstrations themselves are imperfect. In this work we advance offline imitation learning by enabling robots to extrapolate what the human teacher meant, instead of only considering what the human actually showed. We achieve this by hypothesizing that all of the human's demonstrations are trying to convey a single, consistent policy, while the noise and sub-optimality within their behaviors obfuscates the data and introduces unintentional complexity. To recover the underlying policy and learn what the human teacher meant, we introduce Counter-BC, a generalized version of behavior cloning. Counter-BC expands the given dataset to include actions close to behaviors the human demonstrated (i.e., counterfactual actions that the human teacher could have intended, but did not actually show). During training Counter-BC autonomously modifies the human's demonstrations within this expanded region to reach a simple and consistent policy that explains the underlying trends in the human's dataset. Theoretically, we prove that Counter-BC can extract the desired policy from imperfect data, multiple users, and teachers of varying skill levels. Empirically, we compare Counter-BC to state-of-the-art alternatives in simulated and real-world settings with noisy demonstrations, standardized datasets, and real human teachers. See videos of our work here: https://youtu.be/XaeOZWhTt68

Figures (7)

• Figure 1: Learning from imperfect human demonstrations. (Left) The human teaches a robot arm to play air hockey by showing examples of hitting the puck. (Right) The robot learns a control policy based on the human's data. Within the state-of-the-art, the robot trains its policy offline to exactly mimic the actions demonstrated by the human --- but this approach is fundamentally limited when human teachers make mistakes (like missing the puck). Counter-BC tries to imitate what the human meant, not necessarily what the human showed. More specifically, Counter-BC can modify the human's demonstrations within counterfactual sets in order to match underlying behaviors across the entire dataset. This leads to robots that reach simple, consistent explanations of the human's demonstrations, e.g., learning to hit the puck at every state.
• Figure 2: Visualizing the loss in Equation (\ref{['eq:M7']}) for a single state. Here the actions are two dimensional (i.e., in an $x$-$y$ plane), and the demonstrated human action is at the origin. The counterfactual set lies within the dashed white circle ($\Delta = 0.5$). We show the probability distribution over actions for four different policies $\pi_\theta(a \mid s)$, where lighter regions indicate that the action is more likely. (Far Left) The maximum loss occurs when the policy confidently predicts an action outside the counterfactual. (Far Right) The minimum loss occurs when policy confidently predicts an action inside the counterfactual. Note that the predicted action does not need to be the demonstrated action; specifying any action within $\mathcal{C}(s, a)$ can minimize the proposed loss.
• Figure 3: Recovering the underlying function with Counter-BC. In a $1$-dimensional environment the simulated human noisily demonstrates the absolute value function $a = |s| - 0.5 + \epsilon$, where $\epsilon \sim \mathcal{U}(-0.5, +0.5)$. We plot the robot's learned policy averaged across $50$ runs; shaded regions show the standard deviation. With behavior cloning (BC) the robot tries to precisely match the noisy data, resulting in a more complex explanation of the human's demonstrations. (From left to right) Counter-BC with different values of hyperparameter $\Delta$. Increasing $\Delta$ causes Counter-BC to recover increasingly simple explanations of the data by reasoning over larger counterfactual sets, but the resulting policies may diverge from the human's examples.
• Figure 4: Learning from real and simulated humans (see Sections \ref{['sec:S1']} and \ref{['sec:S2']}). Every column shows a different environment, and the rows correspond to demonstrations from real humans, or synthetic operators with uniform, Gaussian, and random noise. Plots illustrate the performance of the policy learned by each algorithm as a function of the number of demonstrated state-action pairs (higher reward or success is better). For Robomimic we tested the Can task with standardized demonstrations from the Multi-Human dataset mandlekar2021matters. Results are averaged across $50$ runs; shaded regions show standard error.
• Figure 5: Learning from increasingly noisy demonstrations (see Section \ref{['sec:S3']}). To regulate the amount of noise, we simulated a human teacher. This teacher selected actions $a = a^* + \epsilon$, where $a^*$ is the optimal action and $\epsilon \sim \mathcal{U}(-\sigma, \sigma)$ is uniform noise. Increasing $\sigma$ led to more noisy, imperfect, and suboptimal human teaching. The policies learned by each algorithm degrade as $\sigma$ increases, but Counter-BC is more robust than the baselines. Results are averaged across $50$ runs with demonstrations containing $400$ state-action pairs.