A Light and Smart Wearable Platform with Multimodal Foundation Model for Enhanced Spatial Reasoning in People with Blindness and Low Vision

TL;DR

The work addresses the navigation and spatial understanding challenges faced by people with blindness and low vision by fine-tuning a multimodal foundation model to incorporate spatial reasoning (LVS-LLaVA) and integrating it into a lightweight wearable platform. A dedicated LVSQA dataset complements the model, and a compact hardware stack with a glasses attachment enables real-time, spatially aware feedback. Across LVSQA and VizWiz evaluations, the approach yields strong spatial reasoning while maintaining general multimodal capabilities, demonstrated via quantitative metrics and ablation comparisons against GPT-4 and LLaVA. The resulting end-to-end system—hardware, model, and dataset—promises enhanced independence for visually impaired users and motivates further work on quantitative benchmarks, UI refinement, and scalable deployment.

Abstract

People with blindness and low vision (pBLV) face significant challenges, struggling to navigate environments and locate objects due to limited visual cues. Spatial reasoning is crucial for these individuals, as it enables them to understand and interpret the spatial relationships in their surroundings, enhancing their ability to navigate and interact more safely and independently. Current multi-modal large language (MLLM) models for low vision people lack the spatial reasoning capabilities needed to effectively assist in these tasks. Moreover, there is a notable absence of lightweight, easy-to-use systems that allow pBLV to effectively perceive and interact with their surrounding environment. In this paper, we propose a novel spatial enhanced multi-modal large language model based approach for visually impaired individuals. By fine-tuning the MLLM to incorporate spatial reasoning capabilities, our method significantly improves the understanding of environmental context, which is critical for navigation and object recognition. The innovation extends to a hardware component, designed as an attachment for glasses, ensuring increased accessibility and ease of use. This integration leverages advanced VLMs to interpret visual data and provide real-time, spatially aware feedback to the user. Our approach aims to bridge the gap between advanced machine learning models and practical, user-friendly assistive devices, offering a robust solution for visually impaired users to navigate their surroundings more effectively and independently. The paper includes an in-depth evaluation using the VizWiz dataset, demonstrating substantial improvements in accuracy and user experience. Additionally, we design a comprehensive dataset to evaluate our method's effectiveness in realworld situations, demonstrating substantial improvements in accuracy and user experience.

Figures (6)

• Figure 1: Overview of our proposed system: On the left, the lightweight and easy-to-use camera designed to be mounted on standard glasses. On the right, the fine-tuned multi-modal large language model (MLLM) enhanced with spatial reasoning capabilities for low vision assistance.
• Figure 2: Flow of our proposed system. Given the observation captured by the camera and a user question, our proposed system use a fine-tuned Low Vision Spatial LLaVA model, which incorporates enhanced spatial reasoning capabilities. Together with specialized prompt engineering tailored for pBLV, the system generates comprehensive answers, effectively addressing the user's query based on surronding environment.
• Figure 3: Examples from the proposed LVSQA dataset, featuring three categories: distance estimation, spatial navigation, and spatial relationships.
• Figure 4: The overview of the system workflow. User Interaction diagram depicts how the user interacts with our system's backend through ESP32S3.
• Figure 5: Examples of our system's smartphone application, demonstrating support for both text and audio formats in question and answer interactions.