Conformalized Teleoperation: Confidently Mapping Human Inputs to High-Dimensional Robot Actions

TL;DR

Conformalized Teleoperation tackles the problem of safely mapping low‑DoF human inputs to high‑DoF robot actions by providing principled uncertainty estimates. It trains a quantile‑aware controller and uses Adaptive Conformalized Quantile Regression (ACQR) to calibrate intervals against a target user, ensuring asymptotic $1-\alpha$ coverage and enabling online adaptation. A dedicated uncertainty monitor based on calibrated interval size flags high‑uncertainty states to prompt intervention. Across 2D navigation and 7DOF Kinova tasks, ACQR improves calibration over baselines and yields a practical mechanism to detect and mitigate failures in assistive teleoperation, advancing reliable, uncertainty‑aware human‑robot interaction.

Abstract

Assistive robotic arms often have more degrees-of-freedom than a human teleoperator can control with a low-dimensional input, like a joystick. To overcome this challenge, existing approaches use data-driven methods to learn a mapping from low-dimensional human inputs to high-dimensional robot actions. However, determining if such a black-box mapping can confidently infer a user's intended high-dimensional action from low-dimensional inputs remains an open problem. Our key idea is to adapt the assistive map at training time to additionally estimate high-dimensional action quantiles, and then calibrate these quantiles via rigorous uncertainty quantification methods. Specifically, we leverage adaptive conformal prediction which adjusts the intervals over time, reducing the uncertainty bounds when the mapping is performant and increasing the bounds when the mapping consistently mis-predicts. Furthermore, we propose an uncertainty-interval-based mechanism for detecting high-uncertainty user inputs and robot states. We evaluate the efficacy of our proposed approach in a 2D assistive navigation task and two 7DOF Kinova Jaco tasks involving assistive cup grasping and goal reaching. Our findings demonstrate that conformalized assistive teleoperation manages to detect (but not differentiate between) high uncertainty induced by diverse preferences and induced by low-precision trajectories in the mapping's training dataset. On the whole, we see this work as a key step towards enabling robots to quantify their own uncertainty and proactively seek intervention when needed.

Figures (11)

• Figure 1: Conformalized Teleoperation. We leverage conformal methods to quantify if the robot's learned controller can reliably lift the human's low-DoF input (joystick) to their desired high-DoF action (7 joint velocities). For any joystick input at the current state, the robot can assess its uncertainty in remapping that input (dot size in the expanded view is proportional to uncertainty at that coordinate). Arrows emphasize directional joystick input. (left, top) If the human pushes up or to the left on the joystick, the robot has low uncertainty, because it knows with high probability the person wants to go forward towards the object. (left, bottom) If the human pushes backwards on the joystick, the robot predicts a large pivot backwards, but is rightfully uncertain this is what the human intended.
• Figure 2: Approach for Conformalized Teleoperation (left) During training, we modify the teleoperation controller $f_\theta$ to regress both the high-dimensional action corresponding with the low-dimensional human input, but also the empirical quantiles. (right) When deployed around a new user, we can calibrate the model to the user's new dataset distribution. Adaptive Conformal Quantile Regression enables us to enlarge or shrink the predicted quantiles to get coverage of the user's desired high-dimensional action.
• Figure 3: 7DOF cup-grasping with latent preferences. (left) $\mathcal{D}_\mathrm{train}$ exhibits multimodality in grasp. (right) When ACQR calibrates to $\mathcal{D}_\mathrm{calib}^A$. Orange dots indicate timesteps with high uncertainty ($U > \beta_{cup}$). $\lambda_t$ represents the multiplicative factor by which the quantile intervals are expanded. While $\alpha_t$ increases initially, it decreases later as the robot orients itself towards the cup and as it approaches the cup, and uncertainty in the desired high-dimensional action increases. Correspondingly, $\lambda_t$ increases at the points of uncertainty. The orange highlights denote timesteps where the uncertainty is greater than threshold$\beta_{cup}$ and aligns temporally with the orange points on the trajectory. For grasps on the lip, there is higher uncertainty at the start of the trajectory and at the grasp location, due to the variance in lip grasp demonstrations in the training dataset.
• Figure 4: 7DOF goal-reaching with latent preferences. (left) $\mathcal{D}_\mathrm{train}$ contains teleoperation trajectories towards red and blue goals. (center) Alice's demonstration contains high uncertainty only at the beginning of the task. (right) Bob's indirect path flags high uncertainty throughout. ACQR maintains lower $\lambda_t$ and higher $\alpha_t$ throughout $\mathcal{D}_\mathrm{calib}^A$ demonstration than throughout $\mathcal{D}_\mathrm{calib}^B$.
• Figure 5: 7DOF goal-reaching (blue) with out-of-distribution low-dimensional input schemes. (Left) The calibration data is collected by querying a mixture of simulated and human users for their desired low-dimensional input for a given high-DOF robot action.(Middle) Our uncertainty monitoring mechanism, at the threshold $\beta_{goal,human}=0.2$, flags high uncertainty for out-of-distribution input schemes $H_1-H_4$.(Right) Calibration using ACQR increases coverage over QR on the calibration data from all in- and out-of-distribution users, increasing coverage towards, but not yet at, the desired level of 90%.