VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

TL;DR

This work tackles the challenge of delivering reliable visual question answering for visually impaired users when images captured by VI individuals are often low-quality or incomplete. It introduces VIAssist, a fine-tuned LLaVA-based system with a CLIP visual encoder and Vicuna-7B, trained via LoRA on a VI-focused instruction dataset to provide actionable retake guidance and accurate answers. The approach yields quantitative gains over baselines (+0.21 in BERTScore and +0.31 in ROUGE) and offers richer, VI-friendly responses that emphasize image quality assessment and targeted shot adjustments. The study highlights the practical impact of adapting multi-modal large language models for VI users and outlines clear future directions, including dataset expansion, automatic reshooting, efficiency improvements, and multi-modal integration to further assistive capabilities.

Abstract

Individuals with visual impairments, encompassing both partial and total difficulties in visual perception, are referred to as visually impaired (VI) people. An estimated 2.2 billion individuals worldwide are affected by visual impairments. Recent advancements in multi-modal large language models (MLLMs) have showcased their extraordinary capabilities across various domains. It is desirable to help VI individuals with MLLMs' great capabilities of visual understanding and reasoning. However, it is challenging for VI people to use MLLMs due to the difficulties in capturing the desirable images to fulfill their daily requests. For example, the target object is not fully or partially placed in the image. This paper explores how to leverage MLLMs for VI individuals to provide visual-question answers. VIAssist can identify undesired images and provide detailed actions. Finally, VIAssist can provide reliable answers to users' queries based on the images. Our results show that VIAssist provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline, respectively.

Figures (9)

• Figure 1: Performance of MLLMs on standard VQA and VI’s VQA datasets.
• Figure 2: In-depth analysis of the MLLMs' performance on VI's VQA dataset.
• Figure 3: Examples of questions and corresponding photos from VI individuals.
• Figure 4: GPT-4V performance on VI individual queries, where the target is completely out of the image.
• Figure 5: System overview of VIAssist.