Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin

TL;DR

The paper presents real-is-sim, a paradigm that uses a continuously synchronized digital twin to bridge the sim-to-real gap in robotic manipulation. By treating the simulator as the execution interface and a 60 Hz visual-physical correction loop as the alignment mechanism, policies are trained on simulator-derived representations and deployed with the real robot as a follower. The approach enables seamless switching between real and virtual evaluation, supports data augmentation and offline policy selection, and demonstrates strong alignment between virtual and real performance on the PushT task, including improved success via offline augmentation. This framework offers a scalable path for safe, efficient policy development and evaluation in real-world robotics.

Abstract

We introduce real-is-sim, a new approach to integrating simulation into behavior cloning pipelines. In contrast to real-only methods, which lack the ability to safely test policies before deployment, and sim-to-real methods, which require complex adaptation to cross the sim-to-real gap, our framework allows policies to seamlessly switch between running on real hardware and running in parallelized virtual environments. At the center of real-is-sim is a dynamic digital twin, powered by the Embodied Gaussian simulator, that synchronizes with the real world at 60Hz. This twin acts as a mediator between the behavior cloning policy and the real robot. Policies are trained using representations derived from simulator states and always act on the simulated robot, never the real one. During deployment, the real robot simply follows the simulated robot's joint states, and the simulation is continuously corrected with real world measurements. This setup, where the simulator drives all policy execution and maintains real-time synchronization with the physical world, shifts the responsibility of crossing the sim-to-real gap to the digital twin's synchronization mechanisms, instead of the policy itself. We demonstrate real-is-sim on a long-horizon manipulation task (PushT), showing that virtual evaluations are consistent with real-world results. We further show how real-world data can be augmented with virtual rollouts and compare to policies trained on different representations derived from the simulator state including object poses and rendered images from both static and robot-mounted cameras. Our results highlight the flexibility of the real-is-sim framework across training, evaluation, and deployment stages. Videos available at https://real-is-sim.github.io.

Figures (10)

• Figure 1: The real-is-sim framework, illustrating the information flow between its components. A policy trained in a physics simulator that can be synchronized with the real world embodiedgaussians controls a simulated robot. Real-world observations continuously update the simulator, maintaining its state close to ground truth. The real robot then mirrors the simulated robot's joint positions. This approach shifts the sim-to-real gap challenge from the policy to the physics simulator.
• Figure 1: Comparison of the real-is-sim framework in the proposed follower configuration (real robot mimics simulation) and an alternative leader configuration (simulation follows real robot). The follower setup requires fewer sim-to-real data streams and maintains identical system connections in both offline and online modes, simplifying deployment.
• Figure 2: Figure adapted from embodiedgaussians. It illustrates how Embodied Gaussians represents the world using physical particles and visual Gaussians. The Gaussians are corrected using real RGB images, and in turn exert fictitious visual forces on the particles to align the simulation with the real world. The adaptation highlights how this mechanism integrates into the real-is-sim framework.
• Figure 2: Tracking failure in two policies, one trained on demonstrations collected with the real robot in the loop, and the other trained entirely in simulation. The left image illustrates a failure case where the simulated physics conflicts with the visual correction required to stay in sync with the real world. Since physics acts as a hard constraint, the correction cannot be applied. Collecting demonstrations in online mode helps avoid such failure modes by encouraging behaviors that maintain alignment between the simulation and reality.
• Figure 3: Comparison of three paradigms: (a) Real-is-Sim, (b) Real-Only, and (c) Sim-to-Real, highlighting their online (real-world interaction) and offline (simulation-based) capabilities. Real-is-Sim offers a unified framework where deploying in a virtual environment is a simplified version of deploying in the real-world, lacking only real-time correction. This ensures seamless transferability. Real-Only is confined to real-world execution. Sim-to-Real struggles with distribution mismatch and dynamics discrepancies, making successful transfer uncertain.