Learning Variable Compliance Control From a Few Demonstrations for Bimanual Robot with Haptic Feedback Teleoperation System

TL;DR

The paper tackles safe and dexterous contact-rich manipulation with rigid robots by marrying a VR-enabled teleoperation interface with haptic feedback to collect demonstrations and a Transformer-based Comp-ACT policy to learn variable Cartesian compliance from few examples. Comp-ACT extends Action Chunking with Transformers to operate in task space, conditioning action chunks on end-effector state and force-torque observations while jointly predicting pose and stiffness $K$ via a CVAE-based policy. The learned policy drives a Forward Dynamics Cartesian Compliance Control (FDCC) loop, enabling safe interaction through time-varying stiffness and reduced contact forces, demonstrated across simulation and five real-world tasks with both single-arm and bimanual setups. Key findings show Comp-ACT achieves high task success and dramatically lowers contact forces compared to ACT, with performance enhanced by F/T observations in a task-dependent manner. This work advances practical imitation learning for compliant, safe manipulation of rigid robots in realistic settings.

Abstract

Automating dexterous, contact-rich manipulation tasks using rigid robots is a significant challenge in robotics. Rigid robots, defined by their actuation through position commands, face issues of excessive contact forces due to their inability to adapt to contact with the environment, potentially causing damage. While compliance control schemes have been introduced to mitigate these issues by controlling forces via external sensors, they are hampered by the need for fine-tuning task-specific controller parameters. Learning from Demonstrations (LfD) offers an intuitive alternative, allowing robots to learn manipulations through observed actions. In this work, we introduce a novel system to enhance the teaching of dexterous, contact-rich manipulations to rigid robots. Our system is twofold: firstly, it incorporates a teleoperation interface utilizing Virtual Reality (VR) controllers, designed to provide an intuitive and cost-effective method for task demonstration with haptic feedback. Secondly, we present Comp-ACT (Compliance Control via Action Chunking with Transformers), a method that leverages the demonstrations to learn variable compliance control from a few demonstrations. Our methods have been validated across various complex contact-rich manipulation tasks using single-arm and bimanual robot setups in simulated and real-world environments, demonstrating the effectiveness of our system in teaching robots dexterous manipulations with enhanced adaptability and safety. Code available at: https://github.com/omron-sinicx/CompACT

Figures (9)

• Figure 1: Proposed teleoperation interface via VR controllers with haptic feedback through vibrations (bottom left) and LfD method Comp-ACT: Compliance Control via Action Chunking with Transformers (bottom right). Our system was evaluated on complex contact-rich tasks on a dual-arm robotic setup (top).
• Figure 2: Teleoperation system for data collection using VR controllers. The controller receives the contact force information from the robot's wrist F/T sensor and uses it to provide haptic feedback to the operator through the rumbling (vibration). The same concept is applied to bimanual tasks where a different VR controller is connected to each arm.
• Figure 3: Button bindings for HTC Vive controller on teleoperation task.
• Figure 4: Comp-ACT's Network architecture. Left: The action sequence, consisting of $n$ robot states (stiffness $K$, target EE's Cartesian pose $x$, and gripper pose $g$), are encoded alongside the current Cartesian pose $x_t$ and the measured F/T $F_t$ by the CVAE encoder. This network is discarded at inference time. Right: The policy inputs are images from multiple viewpoints, the current Cartesian pose, and the measured F/T. The policy predicts a sequence of $n$ future actions.
• Figure 5: Bimanual Wiping task in robosuite: The goal is to pick up the hammer (left), wipe the exposed dirt beneath the hammer with the other arm (center), and then return the hammer to its initial place (right). During demonstrations, The left arm is kept at a constant medium stiffness. In contrast, the right arm begins with a medium stiffness mode to approach the table, then switches to a low stiffness mode to delicately apply force against the table during the wiping motion. The task is randomized by rotating the hammer between $\pm 15\degree$, and changing the shape of the dirt. 30 demonstrations were collected for this task.