Vision Language Models are In-Context Value Learners

TL;DR

This work introduces Generative Value Learning (GVL), a universal value estimator that leverages vision-language models to predict task progress from video. By autoregressively predicting values over shuffled frames and leveraging in-context examples, GVL achieves zero-shot and few-shot accuracy across hundreds of real-world robotic tasks without task-specific training. The approach yields a practical Value Order Correlation metric (VOC) for evaluation, demonstrates strong cross-embodiment and multi-task generalization, and enables downstream uses such as dataset quality assessment, success detection, and advantage-weighted regression. The results suggest that foundation-model supervision can scale visuomotor learning by providing reliable value predictions across diverse settings.

Abstract

Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (\GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advantage-weighted regression -- all without any model training or finetuning.

Figures (17)

• Figure 1: Result highlights. GVL can effectively zero-shot and few-shot predict task progress on diverse and challenging real-world tasks; these capabilities enable expansive set of downstream applications, including dataset filtering, success detection, and policy learning.
• Figure 2: Method overview. Generative Value Learning (GVL) generates values by auto-regressively predicting task completion percentage over shuffled frames, enabling impressive in-context value learning.
• Figure 3: Zero-shot value predictions on OXE datasets. Left: GVL significantly outperforms LIV on datasets with language goals. Right: GVL still outperforms LIV on datasets with image goals despite solving the more difficult task of frame re-shuffling.
• Figure 4: GVL scales up to 250 ALOHA bi-manual tasks and can improve with in-context examples.
• Figure 5: Example GVL predictions on real-world ALOHA tasks. GVL can successfully unshuffle video frames and generate meaningful task values on diverse tasks and camera viewpoints.