Tendon Force Modeling for Sim2Real Transfer of Reinforcement Learning Policies for Tendon-Driven Robots

TL;DR

A method to model the tendon forces produced by typical servo motors, focusing specifically on the transfer of RL policies for a tendon driven finger is proposed, generalizable to different actuators and robot systems, and can enable RL policies to be used widely across tendon systems, advancing capabilities of dexterous manipulators and soft robots.

Abstract

Robots which make use of soft or compliant inter- actions often leverage tendon-driven actuation which enables actuators to be placed more flexibly, and compliance to be maintained. However, controlling complex tendon systems is challenging. Simulation paired with reinforcement learning (RL) could be enable more complex behaviors to be generated. Such methods rely on torque and force-based simulation roll- outs which are limited by the sim-to-real gap, stemming from the actuator and system dynamics, resulting in poor transfer of RL policies onto real robots. To address this, we propose a method to model the tendon forces produced by typical servo motors, focusing specifically on the transfer of RL policies for a tendon driven finger. Our approach extends existing data- driven techniques by leveraging contextual history and a novel data collection test-bench. This test-bench allows us to capture tendon forces undergo contact-rich interactions typical of real- world manipulation. We then utilize our force estimation model in a GPU-accelerated tendon force-driven rigid body simulation to train RL-based controllers. Our transformer-based model is capable of predicting tendon forces within 3% of the maximum motor force and is robot-agnostic. By integrating our learned model into simulation, we reduce the sim-to-real gap for test trajectories by 41%. RL-based controller trained with our model achieves a 50% improvement in fingertip pose tracking tasks on real tendon-driven robotic fingers. This approach is generalizable to different actuators and robot systems, and can enable RL policies to be used widely across tendon systems, advancing capabilities of dexterous manipulators and soft robots.

Figures (10)

• Figure 1: Overview of our pipeline. We start by collecting tendon force data from real spring mass systems and training an estimator. We then use the estimated forces to simulate accurate tendon forces in highly massively parallelization simulator that supports tendon force driven robots. This allows us to train a contact and tendon force aware reinforcement learning controller that is deployable with minimized sim2real gap.
• Figure 2: Tendon-driven system setup for data collection using various springs and finger setup. The system can be represented as a mass spring system. The force sensor detects tendon tension forces applied by the servo motor to the system. For the finger setup, the tendon is routed directly through the load cell, allowing for data collection directly on a component of a dexterous hand.
• Figure 3: Motor motion that extends the spring and comes back to default position. Due to phenomena such as motor stick friction, the forces applied by the motor are non linear and result in high sim to real gap when ideal torque assumption is made in RL setting.
• Figure 4: Finger test bench setup. The two coupled joints are made out of flexible TPU material. The finger is driven by a one tendon. Antagonistic springs are attached to the back of the finger to allow for release mechanism with tendons are not engaged. Example of simulated tendon driven finger. Spring forces are applied at joints. $\alpha$ is an example desired angle used for the RL controller. The force estimated by the model is calculated at control frequency and is applied to multiple consecutive simulation steps.
• Figure 5: Prediction of tendon forces for a weak spring, strong spring and finger setup of a segment of the full testing trajectory corresponding to the 2.0 [s] steps as shown via the minimap. It can be seen that the transformer model manages to generalize to different measurements the best.