RoboReward: General-Purpose Vision-Language Reward Models for Robotics

TL;DR

RoboReward tackles the reward bottleneck in real-world robotic RL by building a large real-robot dataset, RoboReward, augmented with calibrated negative examples, and a standardized evaluation suite, RoboRewardBench. The authors train generalist vision-language reward models at 4B and 8B scale, showing they outperform many larger open-weight baselines on short-horizon tasks and that reward accuracy offline translates to better online RL performance. Real-robot experiments with RoboReward 8B demonstrate meaningful RL improvements and closer proximity to human rewards, though gaps remain. By releasing data, models, and the RoboRewardBench suite, the work provides a practical path forward for developing reliable, general-purpose reward models for robotics.

Abstract

A well-designed reward is critical for effective reinforcement learning-based policy improvement. In real-world robotic domains, obtaining such rewards typically requires either labor-intensive human labeling or brittle, handcrafted objectives. Vision-language models (VLMs) have shown promise as automatic reward models, yet their effectiveness on real robot tasks is poorly understood. In this work, we aim to close this gap by introducing (1) \textbf{RoboReward}, a robotics reward dataset and benchmark built on large-scale real-robot corpora from Open X-Embodiment (OXE) and RoboArena, and (2) vision-language reward models trained on this dataset (RoboReward 4B/8B). Because OXE is success-heavy and lacks failure examples, we propose a \emph{negative examples data augmentation} pipeline that generates calibrated \emph{negatives} and \emph{near-misses} via counterfactual relabeling of successful episodes and temporal clipping to create partial-progress outcomes from the same videos. Using this framework, we produce an extensive training and evaluation dataset that spans diverse tasks and embodiments and enables systematic evaluation of whether state-of-the-art VLMs can reliably provide rewards for robotics. Our evaluation of leading open-weight and proprietary VLMs reveals that no model excels across all tasks, underscoring substantial room for improvement. We then train general-purpose 4B- and 8B-parameter models that outperform much larger VLMs in assigning rewards for short-horizon robotic tasks. Finally, we deploy the 8B-parameter reward VLM in real-robot reinforcement learning and find that it improves policy learning over Gemini Robotics-ER 1.5, a frontier physical reasoning VLM trained on robotics data, by a large margin, while substantially narrowing the gap to RL training with human-provided rewards.

Figures (7)

• Figure 1: We introduce RoboReward, a dataset for training and evaluating general-purpose vision-language reward models for robotics. RoboReward consists of 2,800 real-robot episodes spanning diverse tasks and robots, with human-verified progress scores. In evaluations across 22 proprietary and open-source VLMs, we demonstrate that today's models are lacking in their ability to provide accurate reward feedback for robots. We curate a training dataset of 45,000 scored robot episodes across diverse embodiments and train RoboReward 4B/8B, two general-purpose vision-language reward models for robotics that outperform frontier vision-language models. We open-source all models, training data, and our evaluation benchmark to advance the development of general-purpose reward models for robotics.
• Figure 2: RL performance on three Robomimic tasks using learned reward functions with different reward formulations, averaged over 3 seeds (shaded regions show $\pm$1 standard deviation). Progress-based reward metrics lead to quicker convergence than a binary success metric. Both continuous and discrete progress rewards achieve comparably fast convergence. Thus, we choose discrete progress as the reward type for our benchmark, since it leads to quick convergence and is easier for humans to annotate consistently than continuous progress.
• Figure 3: Strong positive correlation between reward accuracy and downstream RL performance, where the x-axis is maximum MAE minus the model MAE (larger is better; higher reward accuracy).
• Figure 4: Overview of our counterfactual relabeling approach for generating partial success and failure task-video pairs for reward model training and evaluation. Given a successful robot episode, we use a VLM to describe it in detail, and then a sequence of LLM calls to propose alternative instructions for which the same video would result in only partial success or failure scores. A final VLM check verifies the quality of generated labels and rejects invalid labels.
• Figure 5: Real robot tasks: Pick up the brown monkey and move it on top of the yellow towel (Pick-and-place monkey, left) and Pull the drawer out (Open drawer, right). We run real-world RL improvement on each of these tasks, using RoboReward as a reward.