Scaling Sim-to-Real Reinforcement Learning for Robot VLAs with Generative 3D Worlds

Abstract

The strong performance of large vision-language models (VLMs) trained with reinforcement learning (RL) has motivated similar approaches for fine-tuning vision-language-action (VLA) models in robotics. Many recent works fine-tune VLAs directly in the real world to avoid addressing the sim-to-real gap. While real-world RL circumvents sim-to-real issues, it inherently limits the generality of the resulting VLA, as scaling scene and object diversity in the physical world is prohibitively difficult. This leads to the paradoxical outcome of transforming a broadly pretrained model into an overfitted, scene-specific policy. Training in simulation can instead provide access to diverse scenes, but designing those scenes is also costly. In this work, we show that VLAs can be RL fine-tuned without sacrificing generality and with reduced labor by leveraging 3D world generative models. Using these models together with a language-driven scene designer, we generate hundreds of diverse interactive scenes containing unique objects and backgrounds, enabling scalable and highly parallel policy learning. Starting from a pretrained imitation baseline, our approach increases simulation success from 9.7% to 79.8% while achieving a 1.25$\times$ speedup in task completion time. We further demonstrate successful sim-to-real transfer enabled by the quality of the generated digital twins together with domain randomization, improving real-world success from 21.7% to 75% and achieving a 1.13$\times$ speedup. Finally, we further highlight the benefits of leveraging the effectively unlimited data from 3D world generative models through an ablation study showing that increasing scene diversity directly improves zero-shot generalization.

Figures (21)

• Figure 1: Overall pipeline diagram. Real-world data trains an imitation policy $\pi_\textrm{pre}$. Task descriptions are fed to a language-driven scene designer, which forwards layouts to a 3D world generative model to produce digital twin scenes. $\pi_\theta$ is trained across these scenes, initialized from $\pi_\textrm{pre}$, with massive parallelization and domain randomization. Finally, the trained $\pi_\theta$ is deployed in the real world.
• Figure 2: Overview of the simulation environment generation pipeline. A GPT-4o-powered scene designer converts task descriptions into structured scene graphs over semantic roles and spatial relations, which are instantiated into fully interactive 3D worlds. A quality assurance loop filters physically implausible configurations before simulation, enabling scalable, on-demand generation of diverse environments for RL training.
• Figure 3: Architecture diagram of the $\pi_0$ model as a pretrained imitation model $\pi_\textrm{pre}$ (left) and then modified for RL fine-tuning, $\pi_\theta$ (right).
• Figure 4: Experiment overview.
• Figure 5: Training curves for all $N$.