ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting

TL;DR

<3-5 sentence high-level summary> The paper tackles open-world embodied decision-making by bridging spatial reasoning gaps in vision-language models through visual-temporal context prompting. It introduces ROCKET-1, a segmentation-conditioned low-level policy that operates with past segmentation masks and SAM-2 tracking, guided by a hierarchical reasoning system. A backward trajectory relabeling pipeline enables efficient offline training, and a high-level reasoner (GPT-4o/Molmo) provides prompts to ROCKET-1 within a modular, plug-and-play architecture. In Minecraft, the approach yields a $76\%$ absolute improvement in open-world interactions and strong long-horizon task performance, with ablations highlighting the importance of how spatial prompts are fused and which SAM-2 variants are used. The work demonstrates a scalable framework for grounding VLM reasoning into spatially grounded, open-world control, while outlining future directions like ROCKET-2 to handle unseen objects and broader coverage.</3-5 sentence high-level summary>

Abstract

Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to embodied decision-making in open-world environments presents challenges. One critical issue is bridging the gap between discrete entities in low-level observations and the abstract concepts required for effective planning. A common solution is building hierarchical agents, where VLMs serve as high-level reasoners that break down tasks into executable sub-tasks, typically specified using language. However, language suffers from the inability to communicate detailed spatial information. We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2. Our method unlocks the potential of VLMs, enabling them to tackle complex tasks that demand spatial reasoning. Experiments in Minecraft show that our approach enables agents to achieve previously unattainable tasks, with a $\mathbf{76}\%$ absolute improvement in open-world interaction performance. Codes and demos are now available on the project page: https://craftjarvis.github.io/ROCKET-1.

Figures (7)

• Figure 1: Our pipeline solves creative tasks, such as get the obsidian in the original Minecraft version, using the action space identical to human players (mouse and keyboard). We present a novel instruction interface, visual-temporal context prompting, under which we learn a spatial-sensitive policy, ROCKET-1. VLMs identify regions of interest within each observation and guide ROCKET-1 interactions. Different colors in the segmentation represent distinct interaction types, for example, - use, - approach, - switch, - mine block.
• Figure 2: Different pipelines in solving embodied decision-making tasks.(a) End-to-end pipeline modeling token sequences of language, observations, and actions. (b) Language prompting: VLMs decompose instructions for language-conditioned policy execution. (c) Latent prompting: maps discrete behavior tokens to low-level actions. (d) Future-image prompting: fine-tunes VLMs and diffusion models for image-conditioned control. (e) Visual-temporal prompting: VLMs generate segmentations and interaction cues to guide ROCKET-1.
• Figure 3: ROCKET-1 architecture.ROCKET-1 processes observations ($o$), object segmentations ($m$), and interaction types ($c$) to predict actions ($a$) using a causal transformer. Observations and segmentations are concatenated and passed through a visual backbone for deep fusion. Interaction types and segmentations are randomly dropped with a pre-defiened probability during training.
• Figure 4: Trajectory relabeling pipeline in Minecraft. A bounding box and point selection are applied to the image center in the frame preceding the interaction event to identify the interacting object. SAM-2 is then run in reverse temporal order for a specified duration.
• Figure 5: A hierarchical agent structure based on our proposed visual-temporal context prompting. A GPT-4o model decomposes complex tasks into steps based on the current observation, while the Molmo model identifies interactive objects by outputting points. SAM-2 segments these objects based on the point prompts, and ROCKET-1 uses the object masks and interaction types to make decisions. GPT-4o and Molmo run at low frequencies, while SAM-2 and ROCKET-1 operate at the same frequency as the environment.