ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning

TL;DR

ReinboT addresses uneven data quality in vision-language-action robotic systems by integrating dense return maximization into an end-to-end VLA model. It predicts dense returns (ReturnToGo) and leverages max-return sequence modeling to guide long-horizon actions, rather than relying on sparse feedback. On CALVIN mixed-quality data, ReinboT achieves state-of-the-art performance and demonstrates strong few-shot learning and out-of-distribution generalization in real-world tasks. The approach provides a practical path toward more robust and data-efficient embodied AI by coupling RL return maximization with VLA.

Abstract

Vision-Language-Action (VLA) models have shown great potential in general robotic decision-making tasks via imitation learning. However, the variable quality of training data often constrains the performance of these models. On the other hand, offline Reinforcement Learning (RL) excels at learning robust policy models from mixed-quality data. In this paper, we introduce Reinforced robot GPT (ReinboT), a novel end-to-end VLA model that integrates the RL principle of maximizing cumulative reward. ReinboT achieves a deeper understanding of the data quality distribution by predicting dense returns that capture the nuances of manipulation tasks. The dense return prediction capability enables the robot to generate more robust decision-making actions, oriented towards maximizing future benefits. Extensive experiments show that ReinboT achieves state-of-the-art performance on the CALVIN mixed-quality dataset and exhibits superior few-shot learning and out-of-distribution generalization capabilities in real-world tasks.

Figures (16)

• Figure 1: The proposed ReinboT model. We leverage CLIP radford2021learning to encode robot language instructions, utilize ViT alexey2020imagehe2022masked (and perceiver resampler jaegle2021perceiver) to compress and encode the original pixel space of the image state, and utilize MLP to encode the robot proprioception. Moreover, based on the GPT-style transformer radford2018improving, we introduce three prediction token embeddings ([RTG], [ACTION] and [IMAGE]) to predict ReturnToGo, robot action, and future image state respectively. The last layer of hidden features in ReturnToGo decoder is further utilized to predict robot actions. The dense reward in ReturnToGo contains four aspects: sub-goal achievement, task progress, behavior smoothness and task completion.
• Figure 2: (a) Impact of different values of ReturnToGo $\mathcal{L}_{\rm RTG}$ loss weight $\lambda$. (b) Impact of different values of the expectile regression parameter $m$ in the ReturnToGo $\mathcal{L}_{\rm RTG}$ loss function.
• Figure 3: (a) Distribution of ground-truth ReturnToGo of CALVIN mixed-quality training data and distribution of the maximized ReturnToGo predicted by the ReinboT when interacting with the test environment D. (b) Comparison of ReturnToGo in the training data with text annotations in mixed-quality data and the maximized ReturnToGo predicted by the ReinboT at the interaction time step. The impact of different values of the Expectile Regression (ER) parameter $m$ in the $\mathcal{L}_{\rm RTG}$ loss function is investigated.
• Figure 4: Few-shot learning and OOD generalization evaluation scenarios for real-world tasks. Few-shot learning evaluation scenarios include cup grasping (a), bowl grasping and placing (b), and plush toy grasping and placing (c). OOD generalization evaluation scenarios include unseen language instructions (d), desktop backgrounds (e), distractors (f), and manipulated objects (g).
• Figure 5: Distribution of successful realistic trajectories.