Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins

Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters

TL;DR

GaussTwin is proposed, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction and consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines.

Abstract

Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.

GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins

TL;DR

GaussTwin is proposed, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction and consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines.

Abstract

Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.
Paper Structure (19 sections, 9 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 19 sections, 9 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Position-Based Dynamics Simulation
  5. 3D Gaussian Splattings
  6. Methodology
  7. Constraint definition
  8. Scene Initialization
  9. Online Tracking
  10. Experimental Results
  11. Experimental Setup
  12. Datasets
  13. Baselines
  14. Metrics
  15. Results and Analysis
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Tracking performance of GaussTwin on rigid objects and DLO. The system takes multi-view camera observations as input. First, the objects in the scene are masked, represented by particles, and bonded to 3D Gaussians. The motion of these particles is then predicted at each time step using PBD, and subsequently refined through Gaussian splatting optimization. The bottom two rows illustrate the prediction–correction process carried out by GaussTwin.
  • Figure 2: On the left, we show the experimental setup, with the (i) Franka Research 3 robot with the custom end effector tool, (ii) the haptic interface for teleoperation, and (iii) the set of Intel RealSense D415 scene mounted cameras. On the right, we show the set of all objects used for real-robot experiments.
  • Figure 3: We show qualitative results for tracking both the rope and T-shaped block objects. The ground truth, overlaid on the particle simulation, is shown in green, while the simulated spheres are depicted in blue. The arrow indicates the direction of the applied force. Below the particle representation, we show rendered 3D Gaussians throughout the experiment.
  • Figure 4: Error bars showing rope tracking error for five different configurations. The error is measured as the IoU between ground-truth rope pixels and the projected spheres. We compare GaussTwin with two ablations: (i) GaussTwin using only the pose, and (ii) GaussTwin using only the mask.