Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

World-Gymnast: Training Robots with Reinforcement Learning in a World Model

Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, Sherry Yang

TL;DR

World-Gymnast introduces a framework for fine-tuning a vision-language-action policy inside a real-world-trained video world model. By rollouts in WorldGym and VLM-based rewards, it achieves substantial real-robot gains over supervised fine-tuning and software simulators on Bridge/OpenVLA tasks, and demonstrates capabilities such as training on novel language, distractors, test-time adaptation, and iterative policy-world-model refinement. The approach reduces dependence on physical robot data and manual simulators, hinting at scalable paths to household robots, while highlighting challenges from world-model hallucinations and distribution shifts requiring safety considerations. Overall, the work offers a practical, scalable paradigm for data-efficient robot learning that can exploit cloud-based world models and large foundation models to bridge the sim-to-real gap.

Abstract

Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-robot performance. We propose World-Gymnast, which performs RL finetuning of a vision-language-action (VLA) policy by rolling out the policy in an action-conditioned video world model and rewarding the rollouts with a vision-language model (VLM). On the Bridge robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone's household.

World-Gymnast: Training Robots with Reinforcement Learning in a World Model

TL;DR

World-Gymnast introduces a framework for fine-tuning a vision-language-action policy inside a real-world-trained video world model. By rollouts in WorldGym and VLM-based rewards, it achieves substantial real-robot gains over supervised fine-tuning and software simulators on Bridge/OpenVLA tasks, and demonstrates capabilities such as training on novel language, distractors, test-time adaptation, and iterative policy-world-model refinement. The approach reduces dependence on physical robot data and manual simulators, hinting at scalable paths to household robots, while highlighting challenges from world-model hallucinations and distribution shifts requiring safety considerations. Overall, the work offers a practical, scalable paradigm for data-efficient robot learning that can exploit cloud-based world models and large foundation models to bridge the sim-to-real gap.

Abstract

Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-robot performance. We propose World-Gymnast, which performs RL finetuning of a vision-language-action (VLA) policy by rolling out the policy in an action-conditioned video world model and rewarding the rollouts with a vision-language model (VLM). On the Bridge robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone's household.
Paper Structure (59 sections, 6 equations, 9 figures, 6 tables, 1 algorithm)

This paper contains 59 sections, 6 equations, 9 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Markov Decision Process.
  4. Model-Based RL with Foundation Models.
  5. RL with a World Model
  6. Model-Based GRPO with World Model Rollouts
  7. Diverse Training Scenarios in the World Model
  8. Training from Any Frame.
  9. Training on Novel Language Instructions.
  10. Training with Distractions.
  11. Test-Time Training from a Novel Frame
  12. Iterative World Model and Policy Improvement
  13. Experiments
  14. Experimental Setup
  15. Tasks and Pipeline.
  16. ...and 44 more sections

Figures (9)

  • Figure 1: Overview of World-Gymnast. The policy is trained on tasks specified by an initial frame and language instruction. During training, the policy outputs actions which are then passed to the world model (WorldGym quevedo2025worldgymworldmodelenvironment) which generates imagined rollouts. These rollouts are then passed to a VLM which returns a binary task completion reward. This reward is used to update the policy. Once trained, we evaluate the policy on real robots using the AutoEval zhou2025autoeval setup. The resulting real world rollouts (frame-action sequences) from AutoEval can be further used to improve the world model on the particular environment.
  • Figure 2: Qualitative evaluation of policy rollouts in WorldGym with distractors. We compare rollout quality among SFT, World-Gymnast and World-Gymnast-Distract under visual distractions. The task on the left is put blue cup on plate and the SFT policy clearly picks up the wrong cup, while both World-Gymnast variants are able to correctly execute the task. On the right task (put carrot on plate), we can see SFT struggle again and seems to grab the dinosaur along with the carrot. Both World-Gymnast variants are again successful but World-Gymnast-Distract has better grasping and placing movements. It is worth noting that even with the visual artifacts introduced by the imperfect world model, the policies transfer effectively to the real robot setting.
  • Figure 3: Qualitative comparison of rolling out the same action sequence on the real robot from AutoEval zhou2025autoeval, from software simulator SIMPLER li2024evaluating, from WorldGym quevedo2025worldgymworldmodelenvironment, and from World-Gymnast with online world model updates. Rollouts from World-Gymnast adheres more closely to the real world than SIMPLER, suggesting improving the world model through Dyna sutton1991dyna improves the quality of the rollout.
  • Figure 4: Qualitative evaluation of policy rollouts in WorldGym. We compare the World-Gymnast policy fine-tuned with RL and the base policy before performing RL. Left:lift skull; Right:put eggplant in pot.
  • Figure 5: Qualitative evaluation of policy rollouts in AutoEval.(a):close the drawer; (b):open the drawer; (c):put the eggplant in the blue sink; (d):put the eggplant in the yellow basket.
  • ...and 4 more figures