Function based sim-to-real learning for shape control of deformable free-form surfaces

TL;DR

This work tackles the reality gap in shape control for deformable free-form surfaces by introducing a deformation-function–based sim-to-real learning framework that maps simulated geometries to their physical counterparts using sparse marker data. The approach combines an RBF-based spatial warping function with a neural deformation predictor, operating on a compact B-spline descriptor to enable an end-to-end differentiable pipeline for fast, gradient-based inverse kinematics on a pneumatically actuated soft mannequin. Key contributions include (i) a resilient training scheme that uses frames with missing markers, (ii) an end-to-end N_s2r network coupled with a differentiable B-spline decoder and RBF warp, and (iii) a fast IK solver achieving convergence in roughly 15 iterations with sub-0.5 s per-iteration cost, validated against a CAESAR-shape set and 3D scans. The method reduces shape errors by up to ~60% compared with marker-prediction baselines and demonstrates practical applicability to other deformable robots, with implications for accurate, efficient customization in soft robotics and garment fabrication.

Abstract

For the shape control of deformable free-form surfaces, simulation plays a crucial role in establishing the mapping between the actuation parameters and the deformed shapes. The differentiation of this forward kinematic mapping is usually employed to solve the inverse kinematic problem for determining the actuation parameters that can realize a target shape. However, the free-form surfaces obtained from simulators are always different from the physically deformed shapes due to the errors introduced by hardware and the simplification adopted in physical simulation. To fill the gap, we propose a novel deformation function based sim-to-real learning method that can map the geometric shape of a simulated model into its corresponding shape of the physical model. Unlike the existing sim-to-real learning methods that rely on completely acquired dense markers, our method accommodates sparsely distributed markers and can resiliently use all captured frames -- even for those in the presence of missing markers. To demonstrate its effectiveness, our sim-to-real method has been integrated into a neural network-based computational pipeline designed to tackle the inverse kinematic problem on a pneumatically actuated deformable mannequin.

Figures (11)

• Figure 1: Motivation and challenges. (a) A soft robotic mannequin with nine air chambers can be deformed by pneumatic actuation to realize the target body shape of individual customers. Shape approximation errors are visualized by colors. (b) The hardware setup includes a 3D scanner and a motion capture system. The zoom-in view shows the captured markers, where two markers on the belly were missed. (c) Positions of two markers captured while deforming the mannequin.
• Figure 2: The space warping $\mathbf{\Phi}(\cdot)$ is built on the radial basis functions (RBF) with kernel centers located on the free-form surface of a simulated model. Different warping functions need to be determined for different simulated shapes.
• Figure 3: Different sim-to-real strategies by using RBF-based spatial warping -- (a) a marker-prediction based pipeline (the baseline method that can only use frames with complete set of markers) that needs to solve different linear systems for different simulated shapes and (b) our function-prediction based end-to-end pipeline (that can resiliently use all frames -- even for those in the presence of missing markers) where the components circled by dash lines form a sim-to-real network $\mathcal{N}_{s2r}$.
• Figure 4: Steps of generating the compact B-spline representation for the free-form surface of a deformable mannequin.
• Figure 5: NN-based pipeline for forward kinematic computing.