LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics

TL;DR

This work tackles flexible long-horizon household robotics by integrating memory-augmented task planning with an LLM-driven agent orchestration. It deploys three specialized agents (routing, task planning, knowledge base) and memory via Retrieval-Augmented Generation, supported by Grounded SAM and LLaMa3.2-Vision for perception, all running offline without training. Experiments across three household scenarios show strong task planning, improved memory recall with RAG, and a favorable balance between open-source models for specialized versus routing roles. The results suggest that memory-informed, modular LLM systems can enable robust, autonomous household robotics, while highlighting areas for robustness and multimodal integration enhancements.

Abstract

We present an embodied robotic system with an LLM-driven agent-orchestration architecture for autonomous household object management. The system integrates memory-augmented task planning, enabling robots to execute high-level user commands while tracking past actions. It employs three specialized agents: a routing agent, a task planning agent, and a knowledge base agent, each powered by task-specific LLMs. By leveraging in-context learning, our system avoids the need for explicit model training. RAG enables the system to retrieve context from past interactions, enhancing long-term object tracking. A combination of Grounded SAM and LLaMa3.2-Vision provides robust object detection, facilitating semantic scene understanding for task planning. Evaluation across three household scenarios demonstrates high task planning accuracy and an improvement in memory recall due to RAG. Specifically, Qwen2.5 yields best performance for specialized agents, while LLaMA3.1 excels in routing tasks. The source code is available at: https://github.com/marc1198/chat-hsr.

Figures (6)

• Figure 1: Our LLM-driven robotic system autonomously plans tasks and retrieves past interactions to improve object handling, illustrated by LLM-enforced task planning and memory-retrieved reasoning in a household setting.
• Figure 2: The full pipeline, integrating long-horizon task planning. Newly introduced components are highlighted in blue.
• Figure 3: The agent-orchestration architecture
• Figure 4: RAG workflow for long-term question answering: Relevant past actions are retrieved from dialogue history, and the LLM generates responses based on the retrieved context.
• Figure 5: The artificial household environment used in the experiment.