Real-to-Sim Deformable Object Manipulation: Optimizing Physics Models with Residual Mappings for Robotic Surgery

TL;DR

This work tackles the real-to-sim gap in deformable tissue manipulation for robotic surgery by introducing an online residual-mapping module integrated with an extended position-based dynamics (PBD) simulator. It jointly estimates a residual deformation $\Delta_t$ and online per-particle stiffness to improve the simulator's alignment with observations and to enhance predictive accuracy for future tissue deformation. The approach supports both thin-shell and volumetric tissues through geometry-aware residuals and a composite loss that includes data-fit and physical-energy terms plus a smoothness prior, leading to notable reductions in Chamfer distance and improved future deformation prediction. Experimental validation on real tissue with a dVRK demonstrates substantial gains in alignment and forecasting capability, highlighting potential for model-based control and autonomous surgical assistance.

Abstract

Accurate deformable object manipulation (DOM) is essential for achieving autonomy in robotic surgery, where soft tissues are being displaced, stretched, and dissected. Many DOM methods can be powered by simulation, which ensures realistic deformation by adhering to the governing physical constraints and allowing for model prediction and control. However, real soft objects in robotic surgery, such as membranes and soft tissues, have complex, anisotropic physical parameters that a simulation with simple initialization from cameras may not fully capture. To use the simulation techniques in real surgical tasks, the "real-to-sim" gap needs to be properly compensated. In this work, we propose an online, adaptive parameter tuning approach for simulation optimization that (1) bridges the real-to-sim gap between a physics simulation and observations obtained 3D perceptions through estimating a residual mapping and (2) optimizes its stiffness parameters online. Our method ensures a small residual gap between the simulation and observation and improves the simulation's predictive capabilities. The effectiveness of the proposed mechanism is evaluated in the manipulation of both a thin-shell and volumetric tissue, representative of most tissue scenarios. This work contributes to the advancement of simulation-based deformable tissue manipulation and holds potential for improving surgical autonomy.

Figures (7)

• Figure 1: Our proposed online simulation parameter optimization method. It reveals suitable constraints' stiffness parameters for a PBD simulation of a deformable tissue in an online manner. Arrows in purple shows the immediate control direction at the given time step. The tissue's blue regions indicate lower stiffness, whereas red regions have higher stiffness. In this figure, the tissue is initialized with large stiffness parameters.
• Figure 2: A flow chart of the proposed residual mapping module in the simulation loop. At each time step, a control $\mathbf{u_t}$ is applied to both the real tissue and the simulation. In response, the PBD simulation solves for $\mathbf{x_t}$. A perception pipeline processes imagery data to obtain a surface point cloud $\mathbf{z_t}$ of the tissue. The residual mapping module estimates the residual deformation $\Delta_t$ via optimization, which is then used to update the simulation state.
• Figure 3: Visualizations of the proposed residual mapping module for thin-shell and volumetric objects. Purple and blue arrows represent deformation due to optimizing $\mathcal{D}(\cdot)$ and $\mathcal{E}(\cdot)$, respectively. For sub-surface particles that are unobservable to the camera, only $\mathcal{E}(\cdot)$ informs how it will deform.
• Figure 4: The real-to-sim experiment setup in this work. (a): a piece of chicken muscle manipulated by a dVRK manipulator. (b): A perception pipeline estimates depth and a semantic mask of the tissue. (c): surface point cloud is generated by a camera inverse projection. (d): A simulation mesh is created with the initial observation. (e): four manipulation trajectories are visualized with their starting point labeled on real images.
• Figure 5: Comparison of the real-to-sim Chamfer distance (smoothed) between PBD and PBD-RM. In all four trajectories, the residual mapping module significantly reduces the Chamfer distance.