Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware

TL;DR

R2R2R introduces a scalable, hardware-free data-generation pipeline that converts a smartphone-scanned object and a single human demonstration into thousands of photorealistic robot trajectories. By leveraging 3D Gaussian Splatting, part-level segmentation, and differentiable rendering, it generates diverse, robot-proprioceptive–RGB data without dynamics simulation. Two imitation-learning architectures (Diffusion Policy and pi0-FAST) trained on R2R2R data achieve performance comparable to teleoperation across multiple tasks when scaled to 1k trajectories, while delivering significantly higher data throughput. This approach enables accessible, large-scale dexterous manipulation learning across embodiments and object categories, with clear pathways for future physics-informed extensions and broader manipulation regimes.

Abstract

Scaling robot learning requires vast and diverse datasets. Yet the prevailing data collection paradigm-human teleoperation-remains costly and constrained by manual effort and physical robot access. We introduce Real2Render2Real (R2R2R), a novel approach for generating robot training data without relying on object dynamics simulation or teleoperation of robot hardware. The input is a smartphone-captured scan of one or more objects and a single video of a human demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic demonstrations by reconstructing detailed 3D object geometry and appearance, and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to enable flexible asset generation and trajectory synthesis for both rigid and articulated objects, converting these representations to meshes to maintain compatibility with scalable rendering engines like IsaacLab but with collision modeling off. Robot demonstration data generated by R2R2R integrates directly with models that operate on robot proprioceptive states and image observations, such as vision-language-action models (VLA) and imitation learning policies. Physical experiments suggest that models trained on R2R2R data from a single human demonstration can match the performance of models trained on 150 human teleoperation demonstrations. Project page: https://real2render2real.com

Figures (26)

• Figure 1: Real2Render2Real generating robot training data for the task of “Put the Mug on the Coffee Maker”. R2R2R takes as input a multi-view object scan and a monocular human demonstration video. R2R2R then synthesizes diverse, domain-randomized robot executions through parallel rendering and outputs paired image-action data for policy training. This pipeline enables scalable learning across tasks and embodiments without teleoperation or object dynamics simulation.
• Figure 2: Data Generation Efficiency and Average Policy Performance Across Manipulation Tasks.(Left) Performance visualization displaying both task-specific outcomes (faint background lines) and cross-task averages (bold lines with error shading) for policies trained on real (1 human teleoperator) vs. synthetic data (1 human, 1 GPU). The points labeled by demonstration count (50-1000) highlight the scaling in performance and R2R2R's significant throughput advantage, with individual task trajectories illustrating the variance across different manipulation scenarios. (Right) Log-log scale comparison showing data generation throughput between R2R2R (1-100 GPUs) and human teleoperation (1-100 operators) over a 12-hour period. R2R2R needs an upfront time of 10 minutes for human to scan the objects, demonstrate the task, reconstruct the objects and track their trajectory, where subsequentially no human is involved. On a single NVIDIA 4090 GPU, on average, trajectories will be generated at 27x the speed of a single human teleoperator without needing robot hardware.
• Figure 3: 3D Gaussian Splat Object Reconstructions with part-level segmentations derived from feature-based grouping. Objects are reconstructed and segmented into rigid or articulated components using GARField garfield2024.
• Figure 4: Trajectory Interpolation -- R2R2R adapts object motion to varied start/end configurations via spatial normalization and Slerp.
• Figure 5: Physical Experiments Comparing Real2Render2Real to Human Teleoperation Data Efficiency Task success rate is plotted against data generation time in hours. Solid lines represent performance averaged across $\pi_0$-FAST and Diffusion Policy. The Real2Render2Real line (blue square) includes points corresponding to 50, 100, 150, and 1000 trajectories generated by a single Nvidia RTX 4090. The Human Teleoperation line (gold square) includes points corresponding to 50, 100, and 150 trajectories. The Real2Render2Real data generation time includes a 10-minute setup cost, while the Human Teleoperation time is based on the real trajectory collection time of 150 demonstrations. Exact numbers for evaluation results can be found in \ref{['sec:eval_results']}.