LookPlanGraph: Embodied Instruction Following Method with VLM Graph Augmentation

TL;DR

LookPlanGraph addresses grounding LM-based planners for embodied instruction following in dynamic environments by introducing a Memory Graph, a Scene Graph Simulator for feasibility checks, and a Graph Augmentation Module powered by a Vision-Language Model. The framework enables continual graph updates as the agent perceives changes, and a dedicated GraSIF dataset provides standardized, graph-based evaluation across multiple environments. Empirical results show competitive performance in static settings and clear advantages in dynamic tasks, with robust real-world validation. The work advances practical embodied AI by combining structured graph representations with perceptual grounding and automated validation to support real-time replanning.

Abstract

Methods that use Large Language Models (LLM) as planners for embodied instruction following tasks have become widespread. To successfully complete tasks, the LLM must be grounded in the environment in which the robot operates. One solution is to use a scene graph that contains all the necessary information. Modern methods rely on prebuilt scene graphs and assume that all task-relevant information is available at the start of planning. However, these approaches do not account for changes in the environment that may occur between the graph construction and the task execution. We propose LookPlanGraph - a method that leverages a scene graph composed of static assets and object priors. During plan execution, LookPlanGraph continuously updates the graph with relevant objects, either by verifying existing priors or discovering new entities. This is achieved by processing the agents egocentric camera view using a Vision Language Model. We conducted experiments with changed object positions VirtualHome and OmniGibson simulated environments, demonstrating that LookPlanGraph outperforms methods based on predefined static scene graphs. To demonstrate the practical applicability of our approach, we also conducted experiments in a real-world setting. Additionally, we introduce the GraSIF (Graph Scenes for Instruction Following) dataset with automated validation framework, comprising 514 tasks drawn from SayPlan Office, BEHAVIOR-1K, and VirtualHome RobotHow. Project page available at https://lookplangraph.github.io .

Figures (4)

• Figure 1: Static planners rely solely on predefined scene graphs, making them ineffective when objects are missing or misplaced. In contrast, dynamic planners can explore the environment in real time and update the scene graph to account for newly discovered objects. This allows them to successfully execute tasks.
• Figure 2: LookPlanGraph Overview: The LookPlanGraph starts with an instruction and a static environment graph (1). A scene memory graph, initially a copy of the starting graph, is processed by the LM with the task description and is also sent to the Scene Graph Simulator (2). The LM suggests an action, which the Simulator checks for feasibility. If feasible, the action changes the environment and updates the SMG (3). For actions requiring visual feedback (e.g., "discover_objects"), the environment sends an egocentric camera view to the VLM (4). The VLM processes this image, along with previously seen objects from the SMG, to generate an augmented subgraph (5), which updates the SMG (6). This cycle (2-6) repeats until the LM decides that the task is complete.
• Figure 3: LookPlanGraph Prompt Structure. The prompt consists of a static section that remains unchanged across all instructions, a dynamic section constructed from the current memory graph and previous actions, and a VLM-specific prompt used for graph augmentation.
• Figure 4: Execution of "Andrew going home. Help him pack his backpack" task in real environment. Despite the absence of relevant items in the initial graph, LookPlanGraph correctly identified candidate assets, invoked the graph augmentation module to add newly detected objects, and generated a plan to pack a cup, notebook, and mouse into the backpack, while leaving unrelated items untouched. The system subsequently issued a "close backpack" command and marked the task as complete.