Towards Context-aware Support for Color Vision Deficiency: An Approach Integrating LLM and AR
Shogo Morita, Yan Zhang, Takuto Yamauchi, Sinan Chen, Jialong Li, Kenji Tei
TL;DR
The paper addresses the challenge of providing context-aware assistance to individuals with color vision deficiency, beyond traditional display adjustments. It proposes an AR-based interface to capture real-world context and a multi-modal LLM reasoning system using Chain-of-Thought prompting to generate concise, context-sensitive support content. Preliminary user studies with two color-vision-deficient participants across five everyday scenarios show correct context inference and practical usefulness, with an average usability rating of 8.5/10. Limitations include potential LLM inaccuracies and the need to validate effectiveness in workplace settings, which motivates future work on prompt engineering and broader evaluations.
Abstract
People with color vision deficiency often face challenges in distinguishing colors such as red and green, which can complicate daily tasks and require the use of assistive tools or environmental adjustments. Current support tools mainly focus on presentation-based aids, like the color vision modes found in iPhone accessibility settings. However, offering context-aware support, like indicating the doneness of meat, remains a challenge since task-specific solutions are not cost-effective for all possible scenarios. To address this, our paper proposes an application that provides contextual and autonomous assistance. This application is mainly composed of: (i) an augmented reality interface that efficiently captures context; and (ii) a multi-modal large language model-based reasoner that serves to cognitize the context and then reason about the appropriate support contents. Preliminary user experiments with two color vision deficient users across five different scenarios have demonstrated the effectiveness and universality of our application.