Vision-Language-Policy Model for Dynamic Robot Task Planning

TL;DR

Bridging natural language commands with autonomous robot execution in unstructured environments is addressed by a Vision-Language-Policy model fine-tuned on real-world data to generate executable policies. The approach combines semantic scene reasoning with predefined behavior primitives to produce hierarchical, JSON-form policies that can be executed and updated in real time, on-device. Key contributions include a two-stage training/deployment pipeline, two LoRA-based fine-tuning strategies, and extensive real-world validation across multiple robot embodiments demonstrating dynamic policy updates and cross-embodiment generalization. The work has practical impact for rapid deployment of autonomous, instruction-driven robots in diverse, unstructured settings.

Abstract

Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics. This requires robots to perceive and reason over the current task scene through multiple modalities, and to plan their behaviors to achieve their intended goals. Traditional robotic task-planning approaches often struggle to bridge low-level execution with high-level task reasoning, and cannot dynamically update task strategies when instructions change during execution, which ultimately limits their versatility and adaptability to new tasks. In this work, we propose a novel language model-based framework for dynamic robot task planning. Our Vision-Language-Policy (VLP) model, based on a vision-language model fine-tuned on real-world data, can interpret semantic instructions and integrate reasoning over the current task scene to generate behavior policies that control the robot to accomplish the task. Moreover, it can dynamically adjust the task strategy in response to changes in the task, enabling flexible adaptation to evolving task requirements. Experiments conducted with different robots and a variety of real-world tasks show that the trained model can efficiently adapt to novel scenarios and dynamically update its policy, demonstrating strong planning autonomy and cross-embodiment generalization. Videos: https://robovlp.github.io/

Figures (9)

• Figure 1: We introduce the Vision-Language-Policy model, designed for dynamic robot task planning. This innovative model surpasses state-of-the-art performance on semantic understanding, scene reasoning and policy updating through fine-tuning. Real-world experiments demonstrate that the model is applicable to various robotic task scenarios and can be deployed locally across different embodiments.
• Figure 2: System Overview. Stage 1 performs post-training of the VLM using real-world interaction data consisting of images, task instructions, and corresponding policies. Stage 2 deploys the VLP model locally and generates structured policies based on semantic input to achieve real-time robot control and self-updating.
• Figure 3: Vision-Language-Policy model internal structure and real-world task scenario.
• Figure 4: Comparison of different model fine-tuning strategies. (a) Vision+Decoder LoRA: LoRA is applied to both the vision encoder and LM decoder. (b) Decoder-only LoRA: The vision encoder is frozen, and LoRA is applied solely to the LM decoder.
• Figure 5: Real-world system setup for VLP model deployment.