A Virtual Mechanical Interaction Layer Enables Resilient Human-to-Robot Object Handovers

TL;DR

This work tackles resilient human-to-robot object handovers under dynamic object pose changes by introducing a Virtual Model Control (VMC) based interaction layer that couples the robot to the hand via virtual springs, dampers, and links. The controller comprises a Main Controller that drives the gripper toward moving targets and an Auxiliary Controller that enforces collision avoidance, with a gripper module triggering the final grasp; forces follow saturated profiles $F_s = F_{s_{max}} \tanh\left(\frac{k_s |\mathbf{p}|}{F_{s_{max}}}\right) \frac{\mathbf{p}}{|\mathbf{p}|}$ and $F_d = c(|\mathbf{p}|) \dot{\mathbf{p}}$ where $c(s) = c_1 + c_2 \tanh(\beta |\mathbf{p}|)$. Augmented reality via a Meta Quest 3 visualizes the robot’s intended grasp and tracks the user’s hand pose to improve bidirectional communication, with hand pose-derived target points guiding grasp attempts. The approach is validated through two handover experiments with varying object motions and starting poses, plus a user study (n=16) comparing AR-enabled and non-AR conditions and two robot profiles, showing high resilience and a strong user preference for the AR-enabled, authoritative configuration. Collectively, the results demonstrate that the virtual interaction layer can safely adapt to dynamic handovers, enabling faster, more intuitive human-robot collaboration with meaningful implications for real-world assistance and manufacturing tasks.

Abstract

Object handover is a common form of interaction that is widely present in collaborative tasks. However, achieving it efficiently remains a challenge. We address the problem of ensuring resilient robotic actions that can adapt to complex changes in object pose during human-to-robot object handovers. We propose the use of Virtual Model Control to create an interaction layer that controls the robot and adapts to the dynamic changes in the handover process. Additionally, we propose the use of augmented reality to facilitate bidirectional communication between humans and robots during handovers. We assess the performance of our controller in a set of experiments that demonstrate its resilience to various sources of uncertainties, including complex changes to the object's pose during the handover. Finally, we performed a user study with 16 participants to understand human preferences for different robot control profiles and augmented reality visuals in object handovers. Our results showed a general preference for the proposed approach and revealed insights that can guide further development in adapting the interaction with the user.

Figures (5)

• Figure 1: (a) Human-to-robot object handover is shaped by continuous human-robot interactions, coordinated through the Interaction Layer. An Augmented Reality device enables human-robot bidirectional communication by sensing the human hand pose and receiving the robot joints states to visualize the robot intent to the human user. (b) Our Virtual Model Control implementation of the interaction layer. The Main Controller Module consists of Gripper-Target Paired Points (target points are translated by an offset $\alpha$) connected to the gripper and the object/hand through Virtual Rigid Links, with Springs and Dampers between each pair. Repulsive Spring Regions form the Auxiliary Controller Module, where one region is placed here in front of the hand and the object. In this implementation, the Gripper Control Module is not controlled by the Virtual Model Controller due to hardware limitations.
• Figure 2: Tuned parameters of the different virtual components used in our virtual model control along with their force profiles: (a) main and auxiliary springs and (b) main dampers. Units are N for force values, Nm for stiffness coefficients, Nsm for damping coefficients, and m for $\sigma_r$.
• Figure 3: Objects used in the experiments and their grasping pose as seen through the augmented reality interface: (left to right) a cardboard box, a banana, a spoon, and a plastic cup.
• Figure 4: Examples of trajectories from the gripper right finger (red) and its corresponding target point (blue) with temporal markers of recorded snapshots, where (a) is a plastic cup translation example from Experiment I with the object being moved after the robot starts moving and (b) is an example from Experiment II highlighting the effect of the Auxiliary Controller Module with a spherical repulsive region in front of the object pushing the gripper above the object between markers 2 and 3 (instead of continuing through a straight line). Robot coordinate system used as the reference frame is shown in photo 1 of (b). Full experimental runs of the presented trajectories are shown in the Supplementary Video.
• Figure 5: Summary of results from the user study experiments highlighting the success rate and completion time, as well as scores of the three surveys used for assessment (NASA TLX, FSS, and SUS).