Communicating Inferred Goals with Passive Augmented Reality and Active Haptic Feedback

TL;DR

The paper addresses the lack of transparency in robot-inferred goals during human-robot collaboration in shared autonomy. It introduces a multimodal interface that combines passive augmented reality visualizations of multiple likely goals with active haptic prompts to solicit informative human demonstrations, all within a Bayesian goal-inference framework. The key contributions include a unified algorithm mapping robot inference to both AR visuals and haptic prompts, Bayesian belief updates over discrete goals with a Boltzmann-rational likelihood, and a mutual-information-based method to select effective prompts. Experimental results show that AR+Haptic improves teaching efficiency and reduces interaction time compared to single-modality baselines, demonstrating the practical value of closing the loop on robot inference for faster and more intuitive human-robot learning.

Abstract

Robots learn as they interact with humans. Consider a human teleoperating an assistive robot arm: as the human guides and corrects the arm's motion, the robot gathers information about the human's desired task. But how does the human know what their robot has inferred? Today's approaches often focus on conveying intent: for instance, upon legible motions or gestures to indicate what the robot is planning. However, closing the loop on robot inference requires more than just revealing the robot's current policy: the robot should also display the alternatives it thinks are likely, and prompt the human teacher when additional guidance is necessary. In this paper we propose a multimodal approach for communicating robot inference that combines both passive and active feedback. Specifically, we leverage information-rich augmented reality to passively visualize what the robot has inferred, and attention-grabbing haptic wristbands to actively prompt and direct the human's teaching. We apply our system to shared autonomy tasks where the robot must infer the human's goal in real-time. Within this context, we integrate passive and active modalities into a single algorithmic framework that determines when and which type of feedback to provide. Combining both passive and active feedback experimentally outperforms single modality baselines; during an in-person user study, we demonstrate that our integrated approach increases how efficiently humans teach the robot while simultaneously decreasing the amount of time humans spend interacting with the robot. Videos here: https://youtu.be/swq_u4iIP-g

Figures (5)

• Figure 1: Human teleoperating a $7$-DoF robot arm. (Top) The robot believes it should place the cup on a shelf, and passively visualizes the likely options in augmented reality. (Middle) But the robot is still uncertain about the right shelf. To elicit informative feedback, the robot actively signals the haptic wristband, and prompts the human to guide the robot either up or down. (Bottom) The human responds by moving down. In augmented reality, the robot shows that it has inferred the bottom shelf is the human's goal.
• Figure 2: Our haptic wristband for delivering active feedback. This device can both gently squeeze the wearer's wrist and render localized vibration patterns. Left: Exploded view of the electronics housing. The microcontroller (1) sits over the battery (2) and squeeze mechanism (3). Three motor drivers responsible for controlling vibration (4) are accessible behind the front panel. Middle-Left: Fully assembled haptic device with cord (5). Middle-Right: Vibrotactors (6) are held in place by printed casings (7). Right: To prompt the human we squeeze their wrist and activate multiple vibrotactors in a pre-defined pattern. For instance, powering the left and right vibrotactors encourages the human to guide the robot either right or left.
• Figure 3: Example robot and human behavior during the Avoiding task. The robot is trying to drop a strawberry on the plate, but does not know how to avoid the obstacle. When users receive feedback on the GUI (top row), they are more tentative, and spend extra time watching and correcting the robot. This leads to higher interaction time, lower teaching efficiency, and lower scores on the distractor task. By contrast, humans with AR+Haptic feedback (bottom row) are free to focus on the distractor task. When the robot needs help they get an active haptic notification, and passively observe how the robot learns from their corrections. This multimodal feedback enables the human to efficiently teach the robot without having to constantly monitor its behavior.
• Figure 4: Objective results from our user study. The study included four tasks, and here we report the aggregated results across all tasks. Asterisks denote statistically significant comparisons ($p < .01$) and error bars show standard error. We found that AR+Haptic reduced interaction time and increased teaching efficiency as compared to the alternatives. There was no significant difference in distractor score between AR+Haptic and Haptic ($p = 0.65$).
• Figure 5: Subjective results from our user study. Colors are consistent with Fig. \ref{['fig:objective']}, and higher ratings indicate agreement (i.e., conveys intent, prompts for help). Overall, participants preferred teaching the robot with multimodal AR+Haptic feedback. We note that some users reported the AR device to be uncomfortable, which may skew their perception of this condition.