Correspondence-Free, Function-Based Sim-to-Real Learning for Deformable Surface Control

TL;DR

This work tackles the sim-to-real gap in controlling deformable free-form surfaces by introducing a correspondence-free, function-based learning approach. It jointly learns a deformation-function space via an RBF warping, parameterized by coefficients predicted from a compact B-spline surface descriptor, and a confidence map to weight partial data during training; this enables alignment of simulated shapes with real-world measurements without explicit point correspondences. The system integrates into a differentiable NN-based forward/inverse kinematics pipeline, enabling fast gradient-based optimization for shape control across four pneumatically actuated soft robots. The method outperforms direct learning and marker-based approaches, demonstrates robustness to incomplete point clouds, and enables practical IK solutions for tasks like 3D printing on curved molds and collision-free motion planning. Overall, the approach provides a general, scalable route to reliable deformable-surface control in soft robotics, with potential for custom-mold manufacturing and safer autonomous manipulation.

Abstract

This paper presents a correspondence-free, function-based sim-to-real learning method for controlling deformable freeform surfaces. Unlike traditional sim-to-real transfer methods that strongly rely on marker points with full correspondences, our approach simultaneously learns a deformation function space and a confidence map -- both parameterized by a neural network -- to map simulated shapes to their real-world counterparts. As a result, the sim-to-real learning can be conducted by input from either a 3D scanner as point clouds (without correspondences) or a motion capture system as marker points (tolerating missed markers). The resultant sim-to-real transfer can be seamlessly integrated into a neural network-based computational pipeline for inverse kinematics and shape control. We demonstrate the versatility and adaptability of our method on both vision devices and across four pneumatically actuated soft robots: a deformable membrane, a robotic mannequin, and two soft manipulators.

Figures (24)

• Figure 1: Motivation and challenges. (a) Significant geometric discrepancies visualized by a color map are observed between simulation and reality on a soft manipulator composed of six chambers and a soft robotic mannequin with nice pneumatic actuation DoFs. (b) When using a 3D scanner to capture dense point clouds of the real shape, conventional sim-to-real approaches fail due to the absence of known correspondences between scanned points and the regions with partially missed points. (c) When using a motion capture (MoCap) system to collect marker positions for sim-to-real correction, markers can be missed due to occlusion caused by large local deformation (e.g., the two missed on the belly). As a result, this frame cannot be used in conventional correspondence-based sim-to-real learning.
• Figure 2: Overview of our correspondence-free sim-to-real learning pipeline: the function based space warping learning and the differentiable alignment based joint training.
• Figure 3: The parameterization and B-spline fitting steps of a free-form surface include (a) cutting, (b) tracing the boundary, (c) flattening the surface mesh (ref. floater2003mean), and (d) fitting to determine the control points. The B-Spline surfaces and their control points of four different soft robots tested in this paper are shown in (e).
• Figure 4: The space warping $\mathbf{\Phi}(\cdot)$ is built on the radial basis functions (RBF) with kernel centers located on the free-form surface of a simulated model. Different warping functions with kernels having the same set of $(u,v)$-parameters are used to correct different simulated shapes.
• Figure 5: When there is a large gap between (a) real and (b) simulated shapes, the RBF based space warping may generate a weird shape as shown in (c). This problem can be effectively solved by introducing the function compatibility requirement as a geometric regularization during training -- see (d) for the sim-to-real result with this additional regularization. The color map illustrates the distribution of geometric errors.