ReplanVLM: Replanning Robotic Tasks with Visual Language Models

TL;DR

The paper addresses the challenge of robust robotic task planning under perception gaps by leveraging a visual-language model–based framework, ReplanVLM. It introduces internal and external error correction mechanisms that validate and revise plans and execution, coupled with a replanning loop to recover from failures. Experimental results on real robots and simulations show high success rates (average ~94.2%), superior to baselines, and demonstrate strong error detection and correction capabilities in open-world tasks. The work advances autonomous planning by integrating perceptual grounding with corrective feedback, with practical implications for reliable robotic manipulation in complex environments.

Abstract

Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this paper proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks. Videos of our experiments are available at https://youtu.be/NPk2pWKazJc.

Figures (7)

• Figure 1: ReplanVLM overview. We propose a ReplanVLM framework, which is composed of an internal error correction mechanism and an external error correction mechanism. The internal error correction mechanism refers to an Inner Bot, which is developed to assess and correct the task plans and codes generated by the Decision Bot. The external error correction mechanism involves an Extral Bot, which is used to determine whether the task is completed. If the task is deemed incomplete, the Extra Bot identifies the cause of the failure and sends it back to the Decision Bot to replan the task.
• Figure 2: An example of the internal error correction mechanism. For the task requirement "I am hungry" in Task 1, the Decision Bot generates plans and codes to grab "food." The Inner Bot recommends specifying the object to be grabbed as an "apple" and then relays this suggestion back to the Decision Bot. Finally, the Decision Bot creates a new and accurate plan.
• Figure 3: An example of the external error correction mechanism. For the task requirement "give me the red cube" in Task 2, the Extra Bot compares the position of the red cube before and after task execution and finds that it has not moved. Then, the Extra Bot sends this information to the Decision Bot, suggesting that the yellow cube be removed first before grabbing the red cube. Based on this feedback, the Decision Bot generates correct plans and codes.
• Figure 4: Overview of the system setup.
• Figure 5: Example prompts in the ReplanVLM framework.