Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning

Emanuele Musumeci, Michele Brienza, Francesco Argenziano, Abdel Hakim Drid, Vincenzo Suriani, Daniele Nardi, Domenico D. Bloisi

TL;DR

The paper tackles the gap between user intent and environmental constraints in robotic planning by marrying LLM-based reasoning with classical planning through bidimensional goal relaxation. It formalizes a two-axis framework—domain shifts and goal relaxations—to ground high-level goals in 3D Scene Graphs while progressively relaxing infeasible tasks into executable ones. The approach, ContextMatters, outperforms state-of-the-art LLM+PDDL baselines and is validated on 141 relaxation-prone tasks, a diverse 10-environment dataset, and a real TIAGo robot deployment. This work advances robust, context-aware planning for embodied agents and highlights the practical value of contextual goal adaptation in real-world robotics.

Abstract

Embodied agents need to plan and act reliably in real and complex 3D environments. Classical planning (e.g., PDDL) offers structure and guarantees, but in practice it fails under noisy perception and incorrect predicate grounding. On the other hand, Large Language Models (LLMs)-based planners leverage commonsense reasoning, yet frequently propose actions that are unfeasible or unsafe. Following recent works that combine the two approaches, we introduce ContextMatters, a framework that fuses LLMs and classical planning to perform hierarchical goal relaxation: the LLM helps ground symbols to the scene and, when the target is unreachable, it proposes functionally equivalent goals that progressively relax constraints, adapting the goal to the context of the agent's environment. Operating on 3D Scene Graphs, this mechanism turns many nominally unfeasible tasks into tractable plans and enables context-aware partial achievement when full completion is not achievable. Our experimental results show a +52.45% Success Rate improvement over state-of-the-art LLMs+PDDL baseline, demonstrating the effectiveness of our approach. Moreover, we validate the execution of ContextMatter in a real world scenario by deploying it on a TIAGo robot. Code, dataset, and supplementary materials are available to the community at https://lab-rococo-sapienza.github.io/context-matters/.

Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning

TL;DR

The paper tackles the gap between user intent and environmental constraints in robotic planning by marrying LLM-based reasoning with classical planning through bidimensional goal relaxation. It formalizes a two-axis framework—domain shifts and goal relaxations—to ground high-level goals in 3D Scene Graphs while progressively relaxing infeasible tasks into executable ones. The approach, ContextMatters, outperforms state-of-the-art LLM+PDDL baselines and is validated on 141 relaxation-prone tasks, a diverse 10-environment dataset, and a real TIAGo robot deployment. This work advances robust, context-aware planning for embodied agents and highlights the practical value of contextual goal adaptation in real-world robotics.

Abstract

Embodied agents need to plan and act reliably in real and complex 3D environments. Classical planning (e.g., PDDL) offers structure and guarantees, but in practice it fails under noisy perception and incorrect predicate grounding. On the other hand, Large Language Models (LLMs)-based planners leverage commonsense reasoning, yet frequently propose actions that are unfeasible or unsafe. Following recent works that combine the two approaches, we introduce ContextMatters, a framework that fuses LLMs and classical planning to perform hierarchical goal relaxation: the LLM helps ground symbols to the scene and, when the target is unreachable, it proposes functionally equivalent goals that progressively relax constraints, adapting the goal to the context of the agent's environment. Operating on 3D Scene Graphs, this mechanism turns many nominally unfeasible tasks into tractable plans and enables context-aware partial achievement when full completion is not achievable. Our experimental results show a +52.45% Success Rate improvement over state-of-the-art LLMs+PDDL baseline, demonstrating the effectiveness of our approach. Moreover, we validate the execution of ContextMatter in a real world scenario by deploying it on a TIAGo robot. Code, dataset, and supplementary materials are available to the community at https://lab-rococo-sapienza.github.io/context-matters/.

Paper Structure

This paper contains 14 sections, 17 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Contextual Task Adaptation
  5. Architecture
  6. Implementation
  7. Experimental Results
  8. Baselines
  9. Dataset
  10. Metrics
  11. Experiments
  12. Discussion
  13. Real robot setup
  14. Conclusion

Figures (4)

  • Figure 1: Our architecture takes as input a 3D Scene Graph armeni20193d of the environment and a task expressed in natural language. Unfeasible goals can be relaxed up to a certain degree into semantically similar ones, computing the corresponding plan. This mimics the capability of humans to change expected outcomes on-the-go depending on the context.
  • Figure 2: Proposed architecture. The shifting operators are highlighted in orange, while the relaxation operator is highlighted in blue.
  • Figure 3: Pipeline employed to deploy ContextMatters on a real robot. From perception we build a 3DSG of the environment, fed into our pipeline with the initial goal. We obtain the feasible goal in output, which is then translated into a sequence of groundable actions, executed by our robot.
  • Figure 4: Results of ContextMatters across our six-domain dataset, evaluated without domain generation and with grounding on the scene graph.