A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting

TL;DR

This work tackles the challenge of producing high-fidelity, interactive digital twins for robotic manipulation that are simultaneously photorealistic, fast to reconstruct, and actionable for planning. It introduces an end-to-end Real-to-Sim-to-Real pipeline built around 3D Gaussian Splatting (3DGS) to rapidly reconstruct scenes from sparse RGB inputs, coupled with visibility-aware semantic fusion to lift 2D masks into coherent 3D labels and a multi-stage geometry refinement that yields planning-ready collision meshes via Alpha Shapes. The framework integrates these assets into Unity with a ROS 2–MoveIt planning loop, enabling collision-aware manipulation validated on a Franka Emika Panda robot, with real-world success rates around 90% and a mean placement error of about 0.83 cm. Key contributions include the unified end-to-end pipeline, the visibility-aware semantic fusion, and the efficient conversion from noisy 3DGS outputs into watertight, planning-ready geometry, enabling a practical and scalable sim-to-real workflow for manipulation in unstructured environments.

Abstract

Developing high-fidelity, interactive digital twins is crucial for enabling closed-loop motion planning and reliable real-world robot execution, which are essential to advancing sim-to-real transfer. However, existing approaches often suffer from slow reconstruction, limited visual fidelity, and difficulties in converting photorealistic models into planning-ready collision geometry. We present a practical framework that constructs high-quality digital twins within minutes from sparse RGB inputs. Our system employs 3D Gaussian Splatting (3DGS) for fast, photorealistic reconstruction as a unified scene representation. We enhance 3DGS with visibility-aware semantic fusion for accurate 3D labelling and introduce an efficient, filter-based geometry conversion method to produce collision-ready models seamlessly integrated with a Unity-ROS2-MoveIt physics engine. In experiments with a Franka Emika Panda robot performing pick-and-place tasks, we demonstrate that this enhanced geometric accuracy effectively supports robust manipulation in real-world trials. These results demonstrate that 3DGS-based digital twins, enriched with semantic and geometric consistency, offer a fast, reliable, and scalable path from perception to manipulation in unstructured environments.

Figures (4)

• Figure 1: The overall pipeline of this framework uses multi-view video input and 3DGS to reconstruct the scene geometry. Grounded-SAM provides semantic masks, which are fused with the 3D projection to form a semantically-aware digital twin. This twin enables collision-aware motion planning for real robot manipulation.
• Figure 2: Integration and validation of the digital twin framework across simulation and reality. The Unity view Fig.\ref{['fig:unity']} shows the high-fidelity, photorealistic digital twin built with 3DGS and integrated with the physics engine. This model generates and validates collision-aware motion plans visualized in the Rviz interface Fig.\ref{['fig:ros']}, which uses simplified geometry for MoveIt planning. The validated plan is then executed by the real Franka Emika robot Fig.\ref{['fig:setting']}, completing the sim-to-real workflow.
• Figure 3: Qualitative efficacy of the point cloud cleaning pipeline. Top: Raw 3DGS point clouds exhibiting floaters and surface fuzziness, which impede precise collision checking. Bottom: Refined geometries after applying our multi-stage filtering (heuristic filtering and DBSCAN). The process effectively removes artifacts and sharpens boundaries, yielding planning-ready digital twins for manipulation tasks.
• Figure 4: Execution sequence of the multi-step rearrangement task in (a) the real world and (b) the digital twin. The robot grasps the blue box and places it on the cardboard box, then grasps the yellow cube and stacks it on the blue box, and finally grasps the toy hammer and places it in the target area. This demonstrates the framework's capability for complex, zero-shot manipulation with proactive planning validated in simulation.