A Large Vision-Language Model based Environment Perception System for Visually Impaired People

TL;DR

The paper presents a mobile-cloud environment perception system for visually impaired users that combines a ViT-based segmentation model with a large vision-language model to produce global scene descriptions and object-level details. By incorporating segmentation results as external knowledge into LVLM prompts, the method mitigates hallucinations and enables a training-free, efficient workflow executed via simple gestures on a wearable device. Technical evaluations on POPE, MME, and LLaVA-QA90 show consistent improvements over baselines, while exploratory user studies demonstrate practical benefits in real-world scenes. The proposed hardware and prompt-design approach offer a scalable path toward more accurate and usable scene understanding for visually impaired people.

Abstract

It is a challenging task for visually impaired people to perceive their surrounding environment due to the complexity of the natural scenes. Their personal and social activities are thus highly limited. This paper introduces a Large Vision-Language Model(LVLM) based environment perception system which helps them to better understand the surrounding environment, by capturing the current scene they face with a wearable device, and then letting them retrieve the analysis results through the device. The visually impaired people could acquire a global description of the scene by long pressing the screen to activate the LVLM output, retrieve the categories of the objects in the scene resulting from a segmentation model by tapping or swiping the screen, and get a detailed description of the objects they are interested in by double-tapping the screen. To help visually impaired people more accurately perceive the world, this paper proposes incorporating the segmentation result of the RGB image as external knowledge into the input of LVLM to reduce the LVLM's hallucination. Technical experiments on POPE, MME and LLaVA-QA90 show that the system could provide a more accurate description of the scene compared to Qwen-VL-Chat, exploratory experiments show that the system helps visually impaired people to perceive the surrounding environment effectively.

Figures (8)

• Figure 1: . (a) The terminal device of our system. (b) The user wearing the system.
• Figure 2: . The interaction procedure of the proposed system. The user long presses the screen first to take a picture of the current scene and the global description of the image will be played. After a long press action, the user could tap the chair area and the name of its class 'Chiar' is played. Then the user could swipe to the flowerpot and the name of its class 'Flowerpot' is played. Note that the volumes differ for objects with different distances. When the user double tapping the flowerpot area he detailed description of the flowerpot is played.
• Figure 3: . The procedure of global description acquisition.
• Figure 4: . Illustration of our prompt in the global description acquisition procedure. Red words are constructed based on the segmentation result, yellow words are the simple, ordinary prompt, and blue words are the default query. These segmented objects are integrated into one single sentence, which, alongside the default query, serves as the input of LLM.
• Figure 5: . The procedure of local description acquisition.