World Guidance: World Modeling in Condition Space for Action Generation

TL;DR

WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline, and demonstrates that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities.

Abstract

Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that our method significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/

Figures (6)

• Figure 1: WoG first incorporates future observations into the action inference pipeline, projecting them into the condition space for action generation. Subsequently, it decouples future observations from the pipeline and simultaneously predicts these future conditions alongside actions, thereby transferring the knowledge of future conditions into the VLA model.
• Figure 2: Overview of WoG. WoG is trained in two stages. In the first stage, future observations encoded by frozen vision foundation models are queried and compressed by a trainable Q-former-based Future Encoder to form condition representations, which, together with VLM-encoded current observations and instructions, are used for action prediction. In the second stage, the encoder and vision models are frozen, and the VLM backbone is trained to align with the conditions while predicting actions.
• Figure 3: Overview of our real-world experiment setup. The figure shows our robotic platform and sensors (left), the execution of the three tasks under the in-distribution setup (middle), and the modifications applied for the out-of-distribution setup (right).
• Figure 4: Performance after training with UMI data. Compared to training solely with robot data, the WoG showed a 42% improvement in performance on the P&P task and a 33% on the Fold task.
• Figure 5: Detailed illustration of the query mechanisms within WoG. The left panel depicts the future encoder, which maintains a set of learnable queries to extract low-dimensional, action-relevant conditions from the features of pretrained vision models. The right panel illustrates the query mechanism for condition prediction during the second training stage. Here, a set of learnable query embeddings performs cross-attention with the last hidden states of the VLM, producing predictive representations that are supervised to align with the target conditions generated by the now-frozen future encoder.