Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Based Agentic Exploration for Robot Navigation & Manipulation with Skill Orchestration

Abu Hanif Muhammad Syarubany, Farhan Zaki Rahmani, Trio Widianto

TL;DR

This work presents an end-to-end ROS-based pipeline that enables a mobile robot to execute natural-language shopping tasks in indoor corridors. A lightweight semantic map, built from signboards and AprilTags, supports a constrained LLM decision layer that selects high-level actions at each junction, while modular low-level controllers perform navigation and manipulation with safety via a local costmap. The approach emphasizes modularity, debuggability, and separation of semantic reasoning from motion execution, demonstrated through end-to-end experiments in Gazebo and a real-world setup to retrieve multiple items. Key contributions include a weakly supervised YOLO detector for perception, a JSON-based semantic mapping scheme, and an LLM-driven policy that operates over the semantic graph with an explicit history log, enabling scalable, human-readable planning and robust execution in cluttered indoor environments.

Abstract

This paper presents an end-to-end LLM-based agentic exploration system for an indoor shopping task, evaluated in both Gazebo simulation and a corresponding real-world corridor layout. The robot incrementally builds a lightweight semantic map by detecting signboards at junctions and storing direction-to-POI relations together with estimated junction poses, while AprilTags provide repeatable anchors for approach and alignment. Given a natural-language shopping request, an LLM produces a constrained discrete action at each junction (direction and whether to enter a store), and a ROS finite-state main controller executes the decision by gating modular motion primitives, including local-costmap-based obstacle avoidance, AprilTag approaching, store entry, and grasping. Qualitative results show that the integrated stack can perform end-to-end task execution from user instruction to multi-store navigation and object retrieval, while remaining modular and debuggable through its text-based map and logged decision history.

LLM-Based Agentic Exploration for Robot Navigation & Manipulation with Skill Orchestration

TL;DR

This work presents an end-to-end ROS-based pipeline that enables a mobile robot to execute natural-language shopping tasks in indoor corridors. A lightweight semantic map, built from signboards and AprilTags, supports a constrained LLM decision layer that selects high-level actions at each junction, while modular low-level controllers perform navigation and manipulation with safety via a local costmap. The approach emphasizes modularity, debuggability, and separation of semantic reasoning from motion execution, demonstrated through end-to-end experiments in Gazebo and a real-world setup to retrieve multiple items. Key contributions include a weakly supervised YOLO detector for perception, a JSON-based semantic mapping scheme, and an LLM-driven policy that operates over the semantic graph with an explicit history log, enabling scalable, human-readable planning and robust execution in cluttered indoor environments.

Abstract

This paper presents an end-to-end LLM-based agentic exploration system for an indoor shopping task, evaluated in both Gazebo simulation and a corresponding real-world corridor layout. The robot incrementally builds a lightweight semantic map by detecting signboards at junctions and storing direction-to-POI relations together with estimated junction poses, while AprilTags provide repeatable anchors for approach and alignment. Given a natural-language shopping request, an LLM produces a constrained discrete action at each junction (direction and whether to enter a store), and a ROS finite-state main controller executes the decision by gating modular motion primitives, including local-costmap-based obstacle avoidance, AprilTag approaching, store entry, and grasping. Qualitative results show that the integrated stack can perform end-to-end task execution from user instruction to multi-store navigation and object retrieval, while remaining modular and debuggable through its text-based map and logged decision history.
Paper Structure (24 sections, 14 figures, 1 table, 4 algorithms)

This paper contains 24 sections, 14 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Object Detection Development
  5. System Architecture Overview
  6. Perception and Semantic Mapping
  7. Signboard and AprilTag perception
  8. Semantic map representation
  9. Historical trajectory logging
  10. LLM-Based Navigation and Task Parsing
  11. Order parsing from natural language
  12. Navigation policy over the semantic map
  13. Low-Level Motion Execution and Recovery
  14. Motion primitives and local costmap controller
  15. Main controller and recovery state
  16. ...and 9 more sections

Figures (14)

  • Figure 1: Camera setup and responsibilities used in the proposed system. The USB camera is used for signboard understanding and store-entry centering, whereas the RealSense depth camera supports local costmap construction, AprilTag detection, and object tracking.
  • Figure 2: Map layout of the corridor-based shopping environment. Junction signboards (arrows, store icons, and AprilTags) support semantic mapping and navigation, and the shaded region on the right marks the pickup point.
  • Figure 4: System-level pipeline of our agentic exploration stack, including perception, localization, mapping, and an LLM reasoning core. The controller layer uses the semantic map and decision history to trigger low-level skills through a finite-state orchestrator.
  • Figure 6: Representative motion strategies used during exploration and semantic-map building, including AprilTag-based approaching, base re-positioning, local-costmap obstacle avoidance, and store-entry alignment. These primitives are triggered by the main controller to reliably reach signboards and entrances while maintaining safe navigation in corridors.
  • Figure 7: System overview during an exploration run. Showing RViz for semantic-map and trajectory visualization alongside the Gazebo simulation.
  • ...and 9 more figures