Dynamic Handover: Throw and Catch with Bimanual Hands

TL;DR

Dynamic Handover addresses the problem of fast, coordinated throw-and-catch with two dexterous hands by training policies in simulation via Multi-Agent Reinforcement Learning and transferring them to real robots. A three-stage pipeline—base MARL training (Stage1), a real-time goal estimator (Stage2), and end-to-end joint fine-tuning (Stage3)—enables robust sim2real transfer, aided by object-trajectory prediction to inform the catcher. The approach leverages domain randomization and a high-DoF bimanual setup (Allegro Hands on XArm) and demonstrates improvements over baselines in both simulation and real-world experiments across multiple object geometries. The findings highlight the value of MARL coordination and predictive goal estimation in mitigating sim2real gaps and enabling dynamic handover tasks with high-speed manipulation. This work advances practical bimanual dynamic manipulation and provides a foundation for safer, contact-free robot interactions in shared workspaces.

Abstract

Humans throw and catch objects all the time. However, such a seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning in simulation and perform Sim2Real transfer to deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning a trajectory prediction model for the object. Such a model can help the robot catcher has a real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at \url{https://binghao-huang.github.io/dynamic_handover/}.

Figures (8)

• Figure 1: We propose Dynamic Handover, a new bimanual dexterous hands system designed for throwing and catching tasks. The system consists of two Allegro Hands, each individually attached to a separate XArm robot, arranged in a facing configuration. Using multi-agent reinforcement learning, we train policies in a simulation environment and subsequently transfer them to the real world.
• Figure 2: Real Robot System: We employ two Allegro Hands, each individually mounted on separate XArm-6 robots, arranged in a face-to-face configuration. We incorporate a RealSense D435 camera for real-time object position tracking, which is oriented towards the working space. We use k prior states in observation.
• Figure 3: Joint End2End Learning: The two agents receive input from both their own observations and the catcher agent additionally receives the predicted catching position. The goal estimator takes past 20 frames of the object's positions as input and predicts the catch goal for each time step. We use a violet ball to represent the pre-defined goal for the throwing. The orange ball represents the predicted goal for the catcher to catch during the throwing task. The blue ball represents the object that is currently been manipulated.
• Figure 4: Training Curves. The plot shows multi-object training curves of our method and 3 baselines.
• Figure 5: Objects Sets. (a) Training objects. (b) Additional objects in evaluation. (c) Real-world objects.