RLVR-World: Training World Models with Reinforcement Learning

TL;DR

<3-5 sentence high-level summary> RLVR-World introduces a post-training reinforcement learning framework with verifiable rewards to directly optimize world models for task-specific transition-prediction metrics, addressing misalignment with traditional MLE objectives. By unifying language and video modalities into a single autoregressive, token-based formulation and leveraging GRPO for stable RL updates, the approach yields substantial improvements on text-game, web-navigation, and robotic-trajectory prediction tasks, often with far fewer gradient steps than conventional pre-training. The work demonstrates practical downstream benefits in policy evaluation and model-predictive control, and provides detailed analyses of repetition mitigation, metric-oriented optimization, and scaling behavior. It also discusses limitations and future directions toward general-purpose, large-scale, task-aligned world models that can generalize across domains.

Abstract

World models predict state transitions in response to actions and are increasingly developed across diverse modalities. However, standard training objectives such as maximum likelihood estimation (MLE) often misalign with task-specific goals of world models, i.e., transition prediction metrics like accuracy or perceptual quality. In this paper, we present RLVR-World, a unified framework that leverages reinforcement learning with verifiable rewards (RLVR) to directly optimize world models for such metrics. Despite formulating world modeling as autoregressive prediction of tokenized sequences, RLVR-World evaluates metrics of decoded predictions as verifiable rewards. We demonstrate substantial performance gains on both language- and video-based world models across domains, including text games, web navigation, and robot manipulation. Our work indicates that, beyond recent advances in reasoning language models, RLVR offers a promising post-training paradigm for enhancing the utility of generative models more broadly. Code, datasets, models, and video samples are available at the project website: https://thuml.github.io/RLVR-World.

Figures (10)

• Figure 1: Training world models with reinforcement learning. (Left) As world models adopt increasingly advanced and scalable architectures, they are typically pre-trained or supervised fine-tuned using surrogate objectives such as maximum likelihood estimation (MLE), which misalign with task-specific prediction metrics. (Right) We propose post-training world models via reinforcement learning with verifiable rewards (RLVR) to directly optimize for these metrics.
• Figure 2: Illustration of RLVR-World framework. World models across various modalities are unified under a sequence modeling formulation, and task-specific prediction metrics serve as verifiable rewards. (Top) Language-based world models predict verbal state transitions in response to verbal actions. (Bottom) Video-based world models, equipped with a visual tokenizer, predict future visual observations conditioned on action vectors.
• Figure 3: Learning curves of video world models on RT-1. Note the significant difference in the $x$-axis scale between the pre-training and post-training stages.
• Figure 4: Model analysis on RT-1. (a) Test-time scaling: best performance among different numbers of generated samples. (b) RL training scaling: learning curves using different group sizes in GRPO. (c) Metric-oriented optimization: RLVR-World trained and tested on different visual metrics.
• Figure 5: Real2Sim policy evaluation.