SyncTwin: Fast Digital Twin Construction and Synchronization for Safe Robotic Grasping

TL;DR

SyncTwin tackles safe robotic grasping in dynamic, partially observed environments by marrying fast RGB-only 3D reconstruction with real-time digital-twin synchronization. It introduces a two-stage pipeline: Stage I builds simulation-ready object assets from limited RGB data using VGGT, and Stage II continuously aligns streaming observations with stored assets through real-time segmentation and colored-ICP, enabling collision-aware planning in a simulator that closes the real-to-sim-to-real loop. A memory bank of complete object assets supports completing partial observations to improve grasp generation and obstacle avoidance, especially under occlusion. Empirical results on a Panda robot show faster asset reconstruction, higher obstacle-avoidance success, and improved grasp performance compared to baselines, validating the practical viability of persistent digital-twin synchronization for real-world manipulation. The work also outlines scalable extensions such as online asset expansion and distributed architectures to further enhance responsiveness in fast-changing scenes.

Abstract

Accurate and safe grasping under dynamic and visually occluded conditions remains a core challenge in real-world robotic manipulation. We present SyncTwin, a digital twin framework that unifies fast 3D scene reconstruction and real-to-sim synchronization for robust and safety-aware grasping in such environments. In the offline stage, we employ VGGT to rapidly reconstruct object-level 3D assets from RGB images, forming a reusable geometry library for simulation. During execution, SyncTwin continuously synchronizes the digital twin by tracking real-world object states via point cloud segmentation updates and aligning them through colored-ICP registration. The updated twin enables motion planners to compute collision-free and dynamically feasible trajectories in simulation, which are safely executed on the real robot through a closed real-to-sim-to-real loop. Experiments in dynamic and occluded scenes show that SyncTwin improves grasp accuracy and motion safety, demonstrating the effectiveness of digital-twin synchronization for real-world robotic execution.

Figures (11)

• Figure 1: SyncTwin enables fast digital twin and synchronization based on efficient 3D scene reconstruction and object tracking, bridging motion planning for sim-to-real execution, which can be applied to dynamic obstacle avoidance, object tracking, and safe grasping under single-view occlusion in real-world environments.
• Figure 2: Framework of the SyncTwin. Stage I reconstructs simulation-ready 3D assets from RGB images using VGGT and SAM2. Multi-view masks are unprojected into point clouds, then denoised, scaled, and meshed into clean object assets stored in a memory bank. Stage II performs real-time object segmentation, pose tracking, and asset-based completion, enabling grasp generation and reactive motion planning in a closed real2sim2real loop. By continuously updating the digital twin and leveraging simulation for decision making, the system ensures safe and adaptive execution under dynamic and partially occluded environments.
• Figure 3: Supporting-plane noise removal mechanism. A virtual light sphere expands from the object center to identify openings and boundary points, enabling filtering of table-plane noise.
• Figure 4: Overview of the camera predictor module. The red solid line indicates that the first frame is persistently stored in the frame memory. The yellow dashed line represents the operation of saving frames from the previous time step into the memory. The blue and green solid lines denote the data flow and processing steps for the current frame. Together, the sliding-window mechanism enables real-time video segmentation and object-level point cloud tracking with temporally consistent memory updates.
• Figure 5: The comparison of processing time given different numbers of input images. The processing time covers both reconstruction and segmentation. The 5 and 10 images do not apply to the baselines because of fail to estimate the camera extrinsic.