Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance

TL;DR

This work presents Value-Guided Policy Steering (V-GPS), a plug-and-play method that enhances pre-trained generalist robotic policies by re-ranking test-time actions using a language-conditioned value function learned via offline RL. It trains a single Q-function on diverse datasets and applies it across multiple embodiments and tasks without fine-tuning policy weights, achieving robust improvements in real-world and simulated robotic manipulation. The approach addresses precision and robustness failures of large-scale policies and demonstrates significant performance gains across five policies and 12 tasks, highlighting practical impact for deploying generalist robots in diverse environments. Limitations include computational overhead and language/object generalization, with future work focused on scaling the value function and expanding data diversity.

Abstract

Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is generally of mixed quality -- not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is to curate only the highest quality examples. It also remains unclear how optimal data from one embodiment is for training on another embodiment. In this paper, we present a general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy. We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improvement on multiple robotic platforms across a total of 12 tasks. Code and videos can be found at: https://nakamotoo.github.io/V-GPS

Figures (5)

• Figure 1: (V-GPS) We introduce Value-Guided Policy Steering (V-GPS), a novel approach that improves the performance of pre-trained generalist robotic policies by re-ranking their actions at deployment time based on a value function learned via offline RL. The same single V-GPS value function can be combined with any off-the-shelf generalist policy in a plug-and-play manner, without the need to fine-tune or access the policy's weights, improving downstream performance across multiple robotic platforms.
• Figure 2: (Failures of Octo) Octo policy encounters failures such as imprecise grasping (first row), dropping the object prematurely (second row), and holding onto the object for too long (third row).
• Figure 3: (Experimental setup) We evaluate our method on 12 tasks in total. In the real-world WidowX robot platform, we study 6 tasks across 3 different scenes. In the SIMPLER simulated evaluation suite, we study 4 tasks on the WidowX platform and 2 tasks on the Google Robot.
• Figure 4: (Qualitative visualizations) V-GPS improves the precision of grasping the slippery object (first row), prevents the policy's default behavior of releasing the object too early (second row) and holding the object for too long (third row). More qualitative results and videos can be found at https://nakamotoo.github.io/V-GPS
• Figure 5: (Model Architecture.) Our value function uses a ResNet-34 image encoder with FiLM language conditioning.