TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulation

TL;DR

TwinAligner introduces a unified Real2Sim2Real framework that jointly addresses visual and dynamic gaps between real-world robotics data and physics-based simulators. It builds a Mesh-GS Digital Twin with editable 3D Gaussian Splatting and SDF meshes to achieve pixel-level visual alignment and accurate collision geometry, and it aligns dynamics via a gradient-free optimization of robot and object physics using Control-Hit-Slide data. The method enables zero-shot Sim2Real policy learning and cross-environment evaluation, demonstrating strong real-world generalization with data-efficient simulation. Quantitative results show improved rendering quality (PSNR), better dynamic alignment (ADD/ADD-S), and high consistency between real and simulated policy performance, indicating potential to accelerate scalable robot learning. Code and data are released on the project site.

Abstract

The robotics field is evolving towards data-driven, end-to-end learning, inspired by multimodal large models. However, reliance on expensive real-world data limits progress. Simulators offer cost-effective alternatives, but the gap between simulation and reality challenges effective policy transfer. This paper introduces TwinAligner, a novel Real2Sim2Real system that addresses both visual and dynamic gaps. The visual alignment module achieves pixel-level alignment through SDF reconstruction and editable 3DGS rendering, while the dynamic alignment module ensures dynamic consistency by identifying rigid physics from robot-object interaction. TwinAligner improves robot learning by providing scalable data collection and establishing a trustworthy iterative cycle, accelerating algorithm development. Quantitative evaluations highlight TwinAligner's strong capabilities in visual and dynamic real-to-sim alignment. This system enables policies trained in simulation to achieve strong zero-shot generalization to the real world. The high consistency between real-world and simulated policy performance underscores TwinAligner's potential to advance scalable robot learning. Code and data will be released on https://twin-aligner.github.io

Figures (7)

• Figure 1: TwinAligner empowers a physics-aware Real2Sim2Real system for policy learning and closed-loop evaluation.
• Figure 2: Overview of TwinAligner. The framework consists of two phases: Real2Sim and Sim2Real. In the Real2Sim phase, both Visual Alignment and Dynamic Alignment between the simulation and the real world are considered. Policies trained on robotic trajectories collected in the simulation created through Real2Sim can directly zero-shot generalize to the real world.
• Figure 3: TwinAligner can be easily adapted to articulated object by combining with monocular articulation estimation method 3DOI qian2023understanding.
• Figure 4: Comparison of geometry reconstruction quality. Our method reconstruct watertight and detailed meshes, while the baseline results contain inaccurate depths, glitches, and holes.
• Figure 5: The effectiveness of our visual-dynamic Real2Sim alignment. For the robot-object interaction trajectories, we compare real-world camera observation with physics simulation and 3DGS rendering results from TwinAligner. Our method strictly aligns visual-dynamic gap at the pixel level.