Act2Goal: From World Model To General Goal-conditioned Policy

TL;DR

Act2Goal addresses long-horizon robotic manipulation by replacing sole reliance on single-step action prediction with a goal-conditioned visual world model (GCWM) that imagines structured mid-course visual trajectories toward a target goal. It introduces Multi-Scale Temporal Hashing (MSTH) to balance dense proximal frames for fine-grained control with sparse distal frames that anchor global task consistency, facilitated by end-to-end cross-attention between world-model features and the action policy. The approach supports reward-free online adaptation through hindsight goal relabeling and LoRA-based finetuning, enabling rapid autonomous improvement on novel tasks without external supervision. Real-robot experiments show substantial gains in success rates from 0.30 to 0.90 on challenging out-of-distribution tasks within minutes, validating that structured visual planning plus multi-scale control yields robust long-horizon manipulation and strong generalization.

Abstract

Specifying robotic manipulation tasks in a manner that is both expressive and precise remains a central challenge. While visual goals provide a compact and unambiguous task specification, existing goal-conditioned policies often struggle with long-horizon manipulation due to their reliance on single-step action prediction without explicit modeling of task progress. We propose Act2Goal, a general goal-conditioned manipulation policy that integrates a goal-conditioned visual world model with multi-scale temporal control. Given a current observation and a target visual goal, the world model generates a plausible sequence of intermediate visual states that captures long-horizon structure. To translate this visual plan into robust execution, we introduce Multi-Scale Temporal Hashing (MSTH), which decomposes the imagined trajectory into dense proximal frames for fine-grained closed-loop control and sparse distal frames that anchor global task consistency. The policy couples these representations with motor control through end-to-end cross-attention, enabling coherent long-horizon behavior while remaining reactive to local disturbances. Act2Goal achieves strong zero-shot generalization to novel objects, spatial layouts, and environments. We further enable reward-free online adaptation through hindsight goal relabeling with LoRA-based finetuning, allowing rapid autonomous improvement without external supervision. Real-robot experiments demonstrate that Act2Goal improves success rates from 30% to 90% on challenging out-of-distribution tasks within minutes of autonomous interaction, validating that goal-conditioned world models with multi-scale temporal control provide structured guidance necessary for robust long-horizon manipulation. Project page: https://act2goal.github.io/

Figures (8)

• Figure 1: Method Overview. The model receives a visual goal (left), imagines how to achieve it via a goal-conditioned world model (top), and executes the planned actions in the real world (right).
• Figure 2: System Overview. We propose Act2Goal, a goal-conditioned policy that integrates a visual world model with multi-scale temporal control to address long-horizon manipulation. After large-scale offline imitation learning, the model shows high performance on seen settings and strong generalization to unseen scenarios. The reward-free online autonomous improvement stage further improve model's performance through rollout-goal relabel-optimize loop.
• Figure 3: Model Architecture. This figure presents the network architecture of Act2Goal model. On the left, multi-view input frames, including current observation and goal, are encoded into latents via a video encoder and concatenated with noisy latents, then refined into MSTH latent frames through Video DiT blocks. On the right, the robot state and multi-scale features from the world model are fed via cross-attention into isomorphic Action DiT blocks, generating MSTH-structured actions.
• Figure 4: Real World Evaluation. This figure illustrates in-domain and out-of-domain test configurations for three real-world tasks: Whiteboard Word Writing, Dessert Plating, and Plug-In Operation. For each task, Head View Goal displays the target, while Model Rollouts shows the robot’s execution process; these setups are used to evaluate the model’s generalization ability using the success rate as the metric.
• Figure 5: Online Autonomous Improvement Scenarios. This figure illustrates four OOD scenarios from the RoboTwin 2.0 benchmark, corresponding to the hard testing modes of Move Can Pot, Pick Dual Bottles, Place Empty Cup, and Place Shoe. These scenarios serve as the testbed for verifying the effectiveness of autonomous improvement.