MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning

TL;DR

MORE advances zero-shot mobile manipulation planning in large, unknown environments by grounding language reasoning in a hierarchical, memory-augmented scene graph that is pruned to task-relevant subgraphs. A high-level LLM planner orchestrates a suite of object-centric subpolicies, with bounded planning to mitigate hallucinations and enable scalable reasoning across indoor and outdoor spaces. The approach yields strong performance on the BEHAVIOR-1K benchmark and demonstrates real-world transfer on a Toyota HSR, outperforming prior LLM- and VLM-based baselines. This work provides a scalable, reproducible framework for long-horizon robotic rearrangement with practical impact for everyday activities in complex environments.

Abstract

Autonomous long-horizon mobile manipulation encompasses a multitude of challenges, including scene dynamics, unexplored areas, and error recovery. Recent works have leveraged foundation models for scene-level robotic reasoning and planning. However, the performance of these methods degrades when dealing with a large number of objects and large-scale environments. To address these limitations, we propose MORE, a novel approach for enhancing the capabilities of language models to solve zero-shot mobile manipulation planning for rearrangement tasks. MORE leverages scene graphs to represent environments, incorporates instance differentiation, and introduces an active filtering scheme that extracts task-relevant subgraphs of object and region instances. These steps yield a bounded planning problem, effectively mitigating hallucinations and improving reliability. Additionally, we introduce several enhancements that enable planning across both indoor and outdoor environments. We evaluate MORE on 81 diverse rearrangement tasks from the BEHAVIOR-1K benchmark, where it becomes the first approach to successfully solve a significant share of the benchmark, outperforming recent foundation model-based approaches. Furthermore, we demonstrate the capabilities of our approach in several complex real-world tasks, mimicking everyday activities. We make the code publicly available at https://more-model.cs.uni-freiburg.de.

Figures (6)

• Figure 1: We present MORE, an efficient model for the task of rearrangement through mobile manipulation. We utilize 3D scene graphs as a logical scene representation manifold that is filtered to obtain task-relevant subgraphs.
• Figure 2: Overview of MORE. Starting in an unexplored environment, we continuously construct a hierarchical scene graph of the environment based on RGB-D data and semantic segmentation. We first build a dense occupancy map and then extract a navigational Voronoi graph. This graph is separated at room borders based on door locations, then clustered and segmented into rooms and outdoor regions. The resulting scene graph is converted to natural language descriptions. To generate a bounded planning problem, we first filter the observed objects by task relevance and then use an LLM as a task planner in the resulting subgraph. The LLM orchestrates navigation, manipulation, and exploration subpolicies, which in turn result in an updated scene representation.
• Figure 3: Language-based scene graph filtering: We employ a filtering prompt that structurally represents all regions and respective objects of the unfiltered scene $\xi$. Next, we employ an LLM in order to identify which objects are task-relevant. Those are returned, yielding a task-informed sub-graph $\xi_{task}$.
• Figure 4: Language-based reasoning: Assuming a filtered sub graph $\xi_{task}$, we provide the LLM with the task description, a skill API, and the filtered set of regions and objects. Based on the high-level action chosen by the LLM, we orchestrate the next policy steps to be executed in order to fulfill the task at hand.
• Figure 5: Real-world experiments. From top left to bottom right: overview of the environment, throwing away a plastic bottle, fetching a book, grasping a bowl, taking milk out of the fridge, placing cereals on the dinner table.