Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Models

TL;DR

This work tackles scalable HitL deployment by deriving a denoising-based uncertainty metric from diffusion policies to decide when operator input is truly beneficial. A Gaussian Mixture Model captures multi-modality in the policy's denoising vectors and a fixed percentile threshold triggers interventions, eliminating the need for training-time expert input. The approach also uses teleoperation data gathered during interventions to fine-tune the diffusion policy, yielding improved autonomy with limited data, validated across simulated distribution shifts, partial observability, and multi-modal actions, as well as real-robot experiments. Results show reduced human interventions and data-efficient performance gains, demonstrating practical potential for scalable HitL in robotics. $Uncertainty(o_t) = D(V^s_t) + \\alpha Var_g(V^s_t)$ is computed from the denoising vector field to guide intervention decisions.

Abstract

Human-in-the-loop (HitL) robot deployment has gained significant attention in both academia and industry as a semi-autonomous paradigm that enables human operators to intervene and adjust robot behaviors at deployment time, improving success rates. However, continuous human monitoring and intervention can be highly labor-intensive and impractical when deploying a large number of robots. To address this limitation, we propose a method that allows diffusion policies to actively seek human assistance only when necessary, reducing reliance on constant human oversight. To achieve this, we leverage the generative process of diffusion policies to compute an uncertainty-based metric based on which the autonomous agent can decide to request operator assistance at deployment time, without requiring any operator interaction during training. Additionally, we show that the same method can be used for efficient data collection for fine-tuning diffusion policies in order to improve their autonomous performance. Experimental results from simulated and real-world environments demonstrate that our approach enhances policy performance during deployment for a variety of scenarios.

Figures (7)

• Figure 1: HitL policies with denoising uncertainty: We propose using denoising uncertainty as a metric for deciding when to request (human) expert assistance. Predicted de-noising vectors in end-effector position space (illustrated here via arrows on end-effector position) are collected in a vector field, whose inter-mode divergence and intra-mode variance are used to compute policy uncertainty; when this measure exceeds a threshold, operator assistance is requested. We also show that ensuing teleoperation data can be used to fine-tune policies, achieving notable performance improvements with minimal additional data.
• Figure 2: Experiments in simulated environments. We consider three scenarios during policy deployment. (a) Distribution shift; (b) Partial observability (c) Action multi-modality.
• Figure 3: Qualitative visualization of predicted uncertainty, with lighter colors indicating higher uncertainty
• Figure 4: Left: Average success rate of fine-tuning the Lift-sim task with different number of human intervention steps. Right: Sensitivity to threshold selection.
• Figure 5: Real robot experiments: we design our experiments to elicit the challenges described in Sec. \ref{['sec:sim-exp']} on a real robot.