Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Glasses: GenAI-driven Glasses with Haptic Feedback for Navigation of Visually Impaired People

Issatay Tokmurziyev, Miguel Altamirano Cabrera, Muhammad Haris Khan, Yara Mahmoud, Dzmitry Tsetserukou

TL;DR

The results show that LLM-Glasses can deliver reliable navigation support in controlled environments and motivate further work on responsiveness and deployment in more complex real-world scenarios.

Abstract

LLM-Glasses is a wearable navigation system which assists visually impaired people by utilizing YOLO-World object detection, GPT-4o-based reasoning, and haptic feedback for real-time guidance. The device translates visual scene understanding into intuitive tactile feedback on the temples, allowing hands-free navigation. Three studies evaluate the system: recognition of 13 haptic patterns with an average recognition rate of 81.3%, VICON-based guidance with predefined paths using haptic cues, and an LLM-guided scene evaluation with decision accuracies of 91.8% without obstacles, 84.6% with static obstacles, and 81.5% with dynamic obstacles. These results show that LLM-Glasses can deliver reliable navigation support in controlled environments and motivate further work on responsiveness and deployment in more complex real-world scenarios.

LLM-Glasses: GenAI-driven Glasses with Haptic Feedback for Navigation of Visually Impaired People

TL;DR

The results show that LLM-Glasses can deliver reliable navigation support in controlled environments and motivate further work on responsiveness and deployment in more complex real-world scenarios.

Abstract

LLM-Glasses is a wearable navigation system which assists visually impaired people by utilizing YOLO-World object detection, GPT-4o-based reasoning, and haptic feedback for real-time guidance. The device translates visual scene understanding into intuitive tactile feedback on the temples, allowing hands-free navigation. Three studies evaluate the system: recognition of 13 haptic patterns with an average recognition rate of 81.3%, VICON-based guidance with predefined paths using haptic cues, and an LLM-guided scene evaluation with decision accuracies of 91.8% without obstacles, 84.6% with static obstacles, and 81.5% with dynamic obstacles. These results show that LLM-Glasses can deliver reliable navigation support in controlled environments and motivate further work on responsiveness and deployment in more complex real-world scenarios.

Paper Structure

This paper contains 11 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. System Architecture
  3. Object Detection Module
  4. Master Module
  5. Language Module
  6. Haptic Feedback Module
  7. Experimental Evaluation
  8. Haptic Pattern Recognition Study
  9. Haptic Feedback Navigation Performance
  10. LLM and Video-Based Navigation
  11. Conclusions and Future Work

Figures (5)

  • Figure 1: (a) LLM-Glasses prototype, highlighting key components: ESP32 CAM and a haptic feedback mechanism with a five-bar linkage design; (b) an experimental setup with the user navigating through an obstacle course with real-time guidance provided by the system.
  • Figure 2: System Architecture of the LLM-Glasses.
  • Figure 3: (a) 3D model of the LLM-Glasses haptic navigation system; (b) the nine haptic feedback patterns used in the user study, illustrating tapping and sliding motions across different regions of the user’s temples. Each pattern provides distinct sensory input designed to aid in directional navigation.
  • Figure 4: Participants' trajectories for the two guiding paths. The orange area represents the $\pm 0.3$ m tolerance zone.
  • Figure 5: Example scenarios: (A) no obstacles, (B) static obstacles, (C) dynamic obstacles.