Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Woofs to Words: Towards Intelligent Robotic Guide Dogs with Verbal Communication

Yohei Hayamizu, David DeFazio, Hrudayangam Mehta, Zainab Altaweel, Jacqueline Choe, Chao Lin, Jake Juettner, Furui Xiao, Jeremy Blackburn, Shiqi Zhang

Abstract

Assistive robotics is an important subarea of robotics that focuses on the well-being of people with disabilities. A robotic guide dog is an assistive quadruped robot that helps visually impaired people in obstacle avoidance and navigation. Enabling language capabilities for robotic guide dogs goes beyond naively adding an existing dialog system onto a mobile robot. The novel challenges include grounding language in the dynamically changing environment and improving spatial awareness for the human handler. To address those challenges, we develop a novel dialog system for robotic guide dogs that uses LLMs to verbalize both navigational plans and scenes. The goal is to enable verbal communication for collaborative decision-making within the handler-robot team. In experiments, we conducted a human study to evaluate different verbalization strategies and a simulation study to assess the efficiency and accuracy in navigation tasks.

From Woofs to Words: Towards Intelligent Robotic Guide Dogs with Verbal Communication

Abstract

Assistive robotics is an important subarea of robotics that focuses on the well-being of people with disabilities. A robotic guide dog is an assistive quadruped robot that helps visually impaired people in obstacle avoidance and navigation. Enabling language capabilities for robotic guide dogs goes beyond naively adding an existing dialog system onto a mobile robot. The novel challenges include grounding language in the dynamically changing environment and improving spatial awareness for the human handler. To address those challenges, we develop a novel dialog system for robotic guide dogs that uses LLMs to verbalize both navigational plans and scenes. The goal is to enable verbal communication for collaborative decision-making within the handler-robot team. In experiments, we conducted a human study to evaluate different verbalization strategies and a simulation study to assess the efficiency and accuracy in navigation tasks.
Paper Structure (25 sections, 7 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robotic Guide Dog Systems
  4. Robot Dialog Systems
  5. Non-robotic Navigation Aids
  6. Problem Settings and Methodology
  7. Problem Settings and Assumptions
  8. Our Dialog System for Robotic Guide Dogs
  9. Plan Verbalization for Task Specification:
  10. Scene Verbalization for Spatial Awareness:
  11. Implementation Details
  12. Safeguard for Dialog Management:
  13. Plan Generation for Navigation:
  14. Real-World Effectiveness Evaluation: Human Study
  15. Participants
  16. ...and 10 more sections

Figures (7)

  • Figure 1: A legally blind person walking with our robotic guide dog system during a participant study.
  • Figure 2: Our system uses human-robot dialog to define a formal service task. An LLM first determines relevant navigable locations, and a task planner generates multiple action sequences (plans) for each candidate. These plans are summarized for the human via plan verbalization, detailing metrics like navigation cost and door openings. After human selection, the robot executes the chosen plan, providing navigation guidance while providing scene verbalization to describe the surroundings.
  • Figure 3: LLM prompt defining the role of our robot guide dog dialog system. Given possibly ambiguous human service tasks in natural language, the LLM must conduct a dialog and select the location that best satisfies the task.
  • Figure 4: An illustrative example of our robotic guide dog assisting a legally blind participant to navigate to a conference room. The sequence shows: (a) The robot verbalizes the generated navigation plans for a user request, and starts navigation after the user chooses one of the plans. (b, c) During navigation, the system provides real-time scene verbalization, such as passing a lobby and entering a corridor. (d) The robot announces that they have arrived at the destination.
  • Figure 5: An example simulated conversation. The handler implicitly states a purpose sampled from <location, purpose> pair. The dialog system suggests relevant locations. After confirming the handler is looking for a bench, the system generates a formal task and concludes the task specification conversation.
  • ...and 2 more figures