Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation

Yinpei Dai, Run Peng, Sikai Li, Joyce Chai

TL;DR

This work extends zero-shot object navigation to interactive, personalized goals (ZIPON) and proposes ORION, a flexible LLM-driven framework that coordinates perception, navigation, and user dialogue in a think-act-ask loop. ORION leverages six robot modules—control, semantic mapping, open-vocabulary detection, exploration, memory, and interaction—under an expanded LLM action space to ground open-language goals and solicit user feedback. Experiments in simulated HM3D environments and real TIAGo robots show that diverse language feedback substantially improves success rates and navigation efficiency, albeit with tradeoffs between task completion and interaction overhead. The findings emphasize the utility and limitations of LLM-based control for open-world, personalized navigation and highlight directions for richer multi-modal, personalized human-robot collaboration.

Abstract

Zero-Shot Object Navigation (ZSON) enables agents to navigate towards open-vocabulary objects in unknown environments. The existing works of ZSON mainly focus on following individual instructions to find generic object classes, neglecting the utilization of natural language interaction and the complexities of identifying user-specific objects. To address these limitations, we introduce Zero-shot Interactive Personalized Object Navigation (ZIPON), where robots need to navigate to personalized goal objects while engaging in conversations with users. To solve ZIPON, we propose a new framework termed Open-woRld Interactive persOnalized Navigation (ORION), which uses Large Language Models (LLMs) to make sequential decisions to manipulate different modules for perception, navigation and communication. Experimental results show that the performance of interactive agents that can leverage user feedback exhibits significant improvement. However, obtaining a good balance between task completion and the efficiency of navigation and interaction remains challenging for all methods. We further provide more findings on the impact of diverse user feedback forms on the agents' performance. Code is available at https://github.com/sled-group/navchat.

Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation

TL;DR

This work extends zero-shot object navigation to interactive, personalized goals (ZIPON) and proposes ORION, a flexible LLM-driven framework that coordinates perception, navigation, and user dialogue in a think-act-ask loop. ORION leverages six robot modules—control, semantic mapping, open-vocabulary detection, exploration, memory, and interaction—under an expanded LLM action space to ground open-language goals and solicit user feedback. Experiments in simulated HM3D environments and real TIAGo robots show that diverse language feedback substantially improves success rates and navigation efficiency, albeit with tradeoffs between task completion and interaction overhead. The findings emphasize the utility and limitations of LLM-based control for open-world, personalized navigation and highlight directions for richer multi-modal, personalized human-robot collaboration.

Abstract

Zero-Shot Object Navigation (ZSON) enables agents to navigate towards open-vocabulary objects in unknown environments. The existing works of ZSON mainly focus on following individual instructions to find generic object classes, neglecting the utilization of natural language interaction and the complexities of identifying user-specific objects. To address these limitations, we introduce Zero-shot Interactive Personalized Object Navigation (ZIPON), where robots need to navigate to personalized goal objects while engaging in conversations with users. To solve ZIPON, we propose a new framework termed Open-woRld Interactive persOnalized Navigation (ORION), which uses Large Language Models (LLMs) to make sequential decisions to manipulate different modules for perception, navigation and communication. Experimental results show that the performance of interactive agents that can leverage user feedback exhibits significant improvement. However, obtaining a good balance between task completion and the efficiency of navigation and interaction remains challenging for all methods. We further provide more findings on the impact of diverse user feedback forms on the agents' performance. Code is available at https://github.com/sled-group/navchat.
Paper Structure (17 sections, 4 figures, 3 tables)

This paper contains 17 sections, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Interactive Personalized Navigation
  4. The ZIPON Task
  5. Proposed ORION Framework
  6. LLM Action Space
  7. Operated Modules
  8. Experiments
  9. Experimental Design
  10. Evaluation with Simulated Environment
  11. Experimental Setup
  12. User Simulator
  13. Evaluation Results
  14. Evaluation with Real Robots
  15. Experimental Setup
  16. ...and 2 more sections

Figures (4)

  • Figure 1: An example of zero-shot interactive personalized navigation. There are three computers in the room never seen by the robot before. The goal is to find Alice's computer. The robot starts by finding the wrong object and needs to communicate with the user and leverage the user feedback to locate the personalized goal.
  • Figure 2: The ORION framework architecture. The LLM makes sequential decisions to operate different modules to search, detect and navigate in the environment and talk with the user.
  • Figure 3: The dialogue examples for different language feedback types. Corrective feedback indicates errors in the robot's object identification. Descriptive feedback details the goal's appearance and status. Landmark feedback points out salient nearby objects. Procedural feedback suggests a rough navigation path to the goal. The content within the dashed box represents the internal think-act-ask processes the LLM undergoes. The content in the coloured boxes denotes the interactions between the user and the robot. Ellipsis marks indicate the omission of intermediate human/robot dialogue contents and LLM internal process for space brevity.
  • Figure 4: Distribution of task success rates based on interaction turns (turn 1-5) in the mixed feedback setting for all compared methods.