"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairments

TL;DR

This work systematically evaluates real-time VideoLLMs for aiding visually impaired individuals, introducing VisAssistDaily as a multilingual benchmark and SafeVid for proactive hazard perception. It compares GPT-4o, Zhipu, and VITA-1.5 in real-world tasks across Basic Skills, Home Life, and Social Life, finding GPT-4o generally superior while revealing language-consistency and robustness gaps in others. A user study with eight participants underscores practical strengths and privacy/speech-recognition concerns, informing design implications. Leveraging SafeVid, the authors fine-tune VITA-1.5 to achieve a substantial boost in proactive risk recognition, demonstrating the potential for safer, hands-free assistive AI in daily life while outlining key challenges and directions for future work.

Abstract

The visually impaired population faces significant challenges in daily activities. While prior works employ vision language models for assistance, most focus on static content and cannot address real-time perception needs in complex environments. Recent VideoLLMs enable real-time vision and speech interaction, offering promising potential for assistive tasks. In this work, we conduct the first study evaluating their effectiveness in supporting daily life for visually impaired individuals. We first conducted a user survey with visually impaired participants to design the benchmark VisAssistDaily for daily life evaluation. Using VisAssistDaily, we evaluate popular VideoLLMs and find GPT-4o achieves the highest task success rate. We further conduct a user study to reveal concerns about hazard perception. To address this, we propose SafeVid, an environment-awareness dataset, and fine-tune VITA-1.5, improving risk recognition accuracy from 25.00% to 76.00%.We hope this work provides valuable insights and inspiration for future research in this field.

Figures (8)

• Figure 1: Overview of the experiment setup and interaction flow with examples of VideoLLM.
• Figure 2: Overview of VisAssistDaily and SafeVid.
• Figure 3: Illustrations for questionnaire items.
• Figure 4: Word clouds for the user survey's open-ended results.
• Figure 5: Example of SafeVid dataset.