Embodied Task Planning via Graph-Informed Action Generation with Large Lanaguage Model

TL;DR

GiG introduces Graph-in-Graph memory to tackle long-horizon embodied planning by coupling a local scene-graph encoder with a global state-transition graph, enabling structure-aware experience retrieval. A GNN encodes SG_t into embeddings that populate OG with trajectory-aware nodes, while a bounded lookahead module grounds action projections in explicit environment transitions. The memory retrieval mechanism selects structurally similar past trajectories to guide current decisions, improving coherence and planning efficiency. Across Robotouille and ALFWorld benchmarks, GiG consistently outperforms baselines with substantial Pass@1 gains and favorable compute trade-offs, demonstrating the value of graph-structured memory for embodied reasoning in long-horizon tasks.

Abstract

While Large Language Models (LLMs) have demonstrated strong zero-shot reasoning capabilities, their deployment as embodied agents still faces fundamental challenges in long-horizon planning. Unlike open-ended text generation, embodied agents must decompose high-level intent into actionable sub-goals while strictly adhering to the logic of a dynamic, observed environment. Standard LLM planners frequently fail to maintain strategy coherence over extended horizons due to context window limitation or hallucinate transitions that violate constraints. We propose GiG, a novel planning framework that structures embodied agents' memory using a Graph-in-Graph architecture. Our approach employs a Graph Neural Network (GNN) to encode environmental states into embeddings, organizing these embeddings into action-connected execution trace graphs within an experience memory bank. By clustering these graph embeddings, the framework enables retrieval of structure-aware priors, allowing agents to ground current decisions in relevant past structural patterns. Furthermore, we introduce a novel bounded lookahead module that leverages symbolic transition logic to enhance the agents' planning capabilities through the grounded action projection. We evaluate our framework on three embodied planning benchmarks-Robotouille Synchronous, Robotouille Asynchronous, and ALFWorld. Our method outperforms state-of-the-art baselines, achieving Pass@1 performance gains of up to 22% on Robotouille Synchronous, 37% on Asynchronous, and 15% on ALFWorld with comparable or lower computational cost.

Figures (13)

• Figure 1: In tree-based decomposition (left), peer sub-goals are structurally blocked until the current node completes, forcing idle waits. In contrast, planning as Graph (right) allows dynamic instantiation of new sub-goals, enabling the agent to interleave tasks and utilize the idle horizon.
• Figure 2: GiG parses the environment observation to build a scene graph, which is encoded by GNN as a structurally-rich embedding. This embedding is fed into an experience fetcher to retrieve structurally similar past memory and detect exploration loops. An LLM agent generates the next action conditioned on current observation, past related experience, current goal, and bounded look-ahead results.
• Figure 3: Visualization of GNN embedding separation among intra-trace and inter-trace scene graphs.
• Figure 4: Average steps on Robotouille synchronous tasks on Qwen3-235B. Red dots indicate the horizon length of each task type. success only: average completion steps of success attempts. all trials: average steps of all attempts.
• Figure 5: Average steps on Robotouille asynchronous tasks on Qwen3-235B. Red dots indicate the average horizon length of each task. success only: average completion steps of success attempts. all trials: average steps of all attempts.