Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairments

TL;DR

Guide-LLM introduces an embodied LLM-based agent that guides people with visual impairments using a text-based topological map to plan global routes in large indoor spaces. The framework combines a central LLM controller with a lightweight textual environmental representation, a vector-embedding localization system, and a low-level planner for robot control. It leverages the LLM's commonsense reasoning for hazard detection and enables personalization of routes, speeds, and interaction styles. In simulated iGibson experiments, Guide-LLM demonstrates robust localization error detection and recovery, hazard awareness, and effective personalization, highlighting its potential to improve safe and autonomous navigation for PVI. Future work targets on-device processing, autonomous map generation, depth sensing, and real-world usability trials.

Abstract

Navigation presents a significant challenge for persons with visual impairments (PVI). While traditional aids such as white canes and guide dogs are invaluable, they fall short in delivering detailed spatial information and precise guidance to desired locations. Recent developments in large language models (LLMs) and vision-language models (VLMs) offer new avenues for enhancing assistive navigation. In this paper, we introduce Guide-LLM, an embodied LLM-based agent designed to assist PVI in navigating large indoor environments. Our approach features a novel text-based topological map that enables the LLM to plan global paths using a simplified environmental representation, focusing on straight paths and right-angle turns to facilitate navigation. Additionally, we utilize the LLM's commonsense reasoning for hazard detection and personalized path planning based on user preferences. Simulated experiments demonstrate the system's efficacy in guiding PVI, underscoring its potential as a significant advancement in assistive technology. The results highlight Guide-LLM's ability to offer efficient, adaptive, and personalized navigation assistance, pointing to promising advancements in this field.

Figures (3)

• Figure 1: Guide-LLM: The embodied agent consists of a text map, LLM, and navigational modules to guide the user to the destination.
• Figure 2: Guide-LLM framework: LLM (green) serves as the central controller, using commonsense reasoning to interpret user queries and interact with various modules (yellow) for decision-making and navigation tasks. Text map (Green) provides a textual representation of the environment used by the path planning module to create route plans. Vector database 1 (blue) stores static embeddings of the environment images, aiding in consistent localization. Vector database 2 (red) stores navigational image embedding that can be updated or deleted based on the agent's requirements.
• Figure 3: Text map (Left): diagram of text map, part of the text map is extracted. Example text map representation (Middle): User asks the agent to navigate to the elevator. Guide-LLM plans a route (red line) and begins guiding. Along the route, a hazard is detected (wet floor sign), Guide-LLM warns the user and suggests an alternative path (green line), and the chat box (right) shows an example communication between Guide-LLM and the user.