Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Human Awareness in Robot Task Planning with Large Language Models

Yuchen Liu, Luigi Palmieri, Sebastian Koch, Ilche Georgievski, Marco Aiello

TL;DR

The paper tackles proactive robot task planning in dynamic environments by integrating humans into a hierarchical 3D scene graph and leveraging Large Language Models to predict future human activities and ground them into formal planning language. It introduces an architecture that encodes humans as scene-graph nodes, uses LLMs to predict human goals with probabilities, and transforms the problem from single-robot planning to a multi-agent setting with humans as planning agents. A key contribution is an algorithm that extracts domain knowledge, accounts for past observations, generates or augments actions when needed, assigns goal states to humans, and solves the resulting multi-agent planning problem with a standard planner. The approach aims to enable proactive, human-aware robot decision-making in dynamic environments, with future work including implementation, evaluation, and benchmarking in photorealistic simulations and real-world scenarios.

Abstract

The recent breakthroughs in the research on Large Language Models (LLMs) have triggered a transformation across several research domains. Notably, the integration of LLMs has greatly enhanced performance in robot Task And Motion Planning (TAMP). However, previous approaches often neglect the consideration of dynamic environments, i.e., the presence of dynamic objects such as humans. In this paper, we propose a novel approach to address this gap by incorporating human awareness into LLM-based robot task planning. To obtain an effective representation of the dynamic environment, our approach integrates humans' information into a hierarchical scene graph. To ensure the plan's executability, we leverage LLMs to ground the environmental topology and actionable knowledge into formal planning language. Most importantly, we use LLMs to predict future human activities and plan tasks for the robot considering the predictions. Our contribution facilitates the development of integrating human awareness into LLM-driven robot task planning, and paves the way for proactive robot decision-making in dynamic environments.

Towards Human Awareness in Robot Task Planning with Large Language Models

TL;DR

The paper tackles proactive robot task planning in dynamic environments by integrating humans into a hierarchical 3D scene graph and leveraging Large Language Models to predict future human activities and ground them into formal planning language. It introduces an architecture that encodes humans as scene-graph nodes, uses LLMs to predict human goals with probabilities, and transforms the problem from single-robot planning to a multi-agent setting with humans as planning agents. A key contribution is an algorithm that extracts domain knowledge, accounts for past observations, generates or augments actions when needed, assigns goal states to humans, and solves the resulting multi-agent planning problem with a standard planner. The approach aims to enable proactive, human-aware robot decision-making in dynamic environments, with future work including implementation, evaluation, and benchmarking in photorealistic simulations and real-world scenarios.

Abstract

The recent breakthroughs in the research on Large Language Models (LLMs) have triggered a transformation across several research domains. Notably, the integration of LLMs has greatly enhanced performance in robot Task And Motion Planning (TAMP). However, previous approaches often neglect the consideration of dynamic environments, i.e., the presence of dynamic objects such as humans. In this paper, we propose a novel approach to address this gap by incorporating human awareness into LLM-based robot task planning. To obtain an effective representation of the dynamic environment, our approach integrates humans' information into a hierarchical scene graph. To ensure the plan's executability, we leverage LLMs to ground the environmental topology and actionable knowledge into formal planning language. Most importantly, we use LLMs to predict future human activities and plan tasks for the robot considering the predictions. Our contribution facilitates the development of integrating human awareness into LLM-driven robot task planning, and paves the way for proactive robot decision-making in dynamic environments.
Paper Structure (10 sections, 3 figures, 1 algorithm)

This paper contains 10 sections, 3 figures, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. 3D Scene Graphs
  4. LLM-based Robot Task and Motion Planning
  5. Human-aware Robot Task Planning
  6. APPROACH
  7. Problem Statement
  8. Architecture
  9. Transforming Human Awareness into Multi-Agent Task Planning Problem
  10. CONCLUSION

Figures (3)

  • Figure 1: An example of the human-aware robot task planning upon the Allensville environment Armeni2019hierachical. A scene graph is pre-built from the environment with the floor, room, and item layers, where humans are modeled as nodes in the item layer. The edges refer to the semantic relationships. Using the scene graph as the environment representation, the human user queries a Large Language Model (LLM) to predict the next possible human activities and the goal states of the humans. An automated task planner generates a task plan considering humans as additional planning agents.
  • Figure 2: The architecture of the approach.
  • Figure 3: Illustration of the Allensville scene and the corresponding 3D scene graph Armeni2019hierachical. The scene graph has three layers: floor (red), room (green), and item (blue) layers. The humans and part of the item nodes are not visualized.