Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots

TL;DR

This work tackles the challenge of offline model-based reinforcement learning for real robotics by introducing RWM-U, an uncertainty-aware autoregressive world model, and MOPO-PPO, an uncertainty-penalized on-policy policy optimizer. By propagating epistemic uncertainty through long-horizon imagined rollouts and penalizing high-uncertainty transitions, the approach enables stable, fully offline learning on real robots and data-rich simulation environments. Empirical results across manipulation and locomotion tasks—including deployments on ANYmal D and Unitree G1—demonstrate improved robustness and greater data efficiency, with real-world data further enhancing performance when balanced with simulated experiences. The findings indicate that principled uncertainty handling can make offline MBRL practical for real-world robotics, enabling reuse of past data without iterative online interaction.

Abstract

Reinforcement Learning (RL) has achieved impressive results in robotics, yet high-performing pipelines remain highly task-specific, with little reuse of prior data. Offline Model-based RL (MBRL) offers greater data efficiency by training policies entirely from existing datasets, but suffers from compounding errors and distribution shift in long-horizon rollouts. Although existing methods have shown success in controlled simulation benchmarks, robustly applying them to the noisy, biased, and partially observed datasets typical of real-world robotics remains challenging. We present a principled pipeline for making offline MBRL effective on physical robots. Our RWM-U extends autoregressive world models with epistemic uncertainty estimation, enabling temporally consistent multi-step rollouts with uncertainty effectively propagated over long horizons. We combine RWM-U with MOPO-PPO, which adapts uncertainty-penalized policy optimization to the stable, on-policy PPO framework for real-world control. We evaluate our approach on diverse manipulation and locomotion tasks in simulation and on real quadruped and humanoid, training policies entirely from offline datasets. The resulting policies consistently outperform model-free and uncertainty-unaware model-based baselines, and fusing real-world data in model learning further yields robust policies that surpass online model-free baselines trained solely in simulation.

Figures (8)

• Figure 1: Overview of RWM-U and MOPO-PPO. RWM-U augments autoregressive world models with ensemble-based uncertainty estimation, quantifying epistemic uncertainty (shade) that arises from limited or biased offline data (left). By propagating this uncertainty consistently over long imagined rollouts, RWM-U identifies regions where predictions become unreliable under distribution shift. This uncertainty signal is then used during policy learning to penalize high-risk imagined transitions by MOPO-PPO (middle), enabling stable long-horizon rollouts and making fully offline model-based reinforcement learning practical on real robots (right).
• Figure 2: Training diagram of RWM-U and MOPO-PPO. RWM-U extends autoregressive world models with ensemble-based uncertainty estimation. Each ensemble member independently predicts a Gaussian distribution over the next observation. The variance within each prediction captures aleatoric uncertainty, while the variance across ensemble means estimates epistemic uncertainty arising from limited or biased offline data. During policy training in imagination, MOPO-PPO penalizes high-risk imagined transitions using this uncertainty signal, balancing task performance against model confidence.
• Figure 3: Autoregressive trajectory prediction (left) and uncertainty estimation (right) by RWM-U. Solid lines represent ground truth, while dashed lines denote predicted state evolution. Predictions commence at $t = 32$ using historical observations, with future observations predicted autoregressively by feeding prior predictions back into the model. The epistemic uncertainty estimate by RWM-U aligns with the long-horizon prediction error and thus sets a reliable metric in policy training.
• Figure 4: Imagination reward (left) and epistemic uncertainty (right) during MOPO-PPO training. The policy evaluation on real dynamics is visualized in dots (left). Small penalties lead to overconfident but unreliable policies, while large penalties result in overly conservative behaviors. A well-calibrated penalty (dark blue) achieves the exploration-exploitation balance.
• Figure 5: Normalized episodic rewards across diverse robotic environments and offline RL algorithms with different dataset types. MOPO-PPO consistently outperforms uncertainty-unaware baselines, particularly in complex locomotion tasks.