Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface

TL;DR

Real2Edit2Real tackles the data-efficiency bottleneck in robotic policy learning by linking 3D geometry editing with 2D multi-view RGB data to synthesize novel, physically consistent demonstrations. The method couples metric-scale geometry reconstruction, depth-aware spatial editing with pose correction, and a depth-conditioned, multi-view video generator to produce realistic, varied trajectories without simulators. Empirically, policies trained on generated data from as few as 1–5 source demos match or exceed those trained on 50 real demos, achieving up to 10–50x data efficiency across four real-world tasks and enabling editing in height and texture. This framework promises a unified, scalable data-generation platform for robotics that preserves visual fidelity and interaction realism while expanding spatial generalization.

Abstract

Recent progress in robot learning has been driven by large-scale datasets and powerful visuomotor policy architectures, yet policy robustness remains limited by the substantial cost of collecting diverse demonstrations, particularly for spatial generalization in manipulation tasks. To reduce repetitive data collection, we present Real2Edit2Real, a framework that generates new demonstrations by bridging 3D editability with 2D visual data through a 3D control interface. Our approach first reconstructs scene geometry from multi-view RGB observations with a metric-scale 3D reconstruction model. Based on the reconstructed geometry, we perform depth-reliable 3D editing on point clouds to generate new manipulation trajectories while geometrically correcting the robot poses to recover physically consistent depth, which serves as a reliable condition for synthesizing new demonstrations. Finally, we propose a multi-conditional video generation model guided by depth as the primary control signal, together with action, edge, and ray maps, to synthesize spatially augmented multi-view manipulation videos. Experiments on four real-world manipulation tasks demonstrate that policies trained on data generated from only 1-5 source demonstrations can match or outperform those trained on 50 real-world demonstrations, improving data efficiency by up to 10-50x. Moreover, experimental results on height and texture editing demonstrate the framework's flexibility and extensibility, indicating its potential to serve as a unified data generation framework.

Figures (22)

• Figure 1: The overview of Real2Edit2Real. Real2Edit2Real generates diverse robotic demonstrations, featuring 10-50$\times$ improvement on data efficiency compared with real-world collection across four tasks.
• Figure 2: The framework of Real2Edit2Real. (1) Reconstruct metric-scale geometry from multi-view observations. (2)Synthesize novel trajectories with reliable depth rendering. (3) Generate demonstrations controlled by temporal depth signals.
• Figure 3: The framework of 3D-Controlled Video Generation. We utilize depth as the 3D control interface, in conjunction with edges, actions, and ray maps, to guide the generation of multi-view demonstrations.
• Figure 4: Experiment setups of (a) height and (b) texture editing.
• Figure 5: Ablation study of geometry reconstruction. The left end-effector is in red, and the right end-effector is in yellow.