Describe Now: User-Driven Audio Description for Blind and Low Vision Individuals

TL;DR

This work proposes user-driven AI-generated audio descriptions that BLV viewers can trigger at will to control timing and detail level, addressing shortcomings of fixed ADs. By pre-generating concise and detailed ADs using GPT-4V and a simple C/D-based interface across seven video genres, the study provides empirical and thematic evidence on how end-user control affects engagement, workload, and content accessibility. Key contributions include quantifying AD request frequency by genre, revealing the trade-offs between control and cognitive load, and outlining design principles for future user-driven AD platforms that integrate multisensory cues and adaptive detail. The findings suggest that enabling BLV users to tailor ADs can broaden the range of accessible content while informing how describers and AI systems collaborate, paving the way for customizable, on-demand descriptions in real-world video platforms.

Abstract

Audio descriptions (AD) make videos accessible for blind and low vision (BLV) users by describing visual elements that cannot be understood from the main audio track. AD created by professionals or novice describers is time-consuming and offers little customization or control to BLV viewers on description length and content and when they receive it. To address this gap, we explore user-driven AI-generated descriptions, enabling BLV viewers to control both the timing and level of detail of the descriptions they receive. In a study, 20 BLV participants activated audio descriptions for seven different video genres with two levels of detail: concise and detailed. Our findings reveal differences in the preferred frequency and level of detail of ADs for different videos, participants' sense of control with this style of AD delivery, and its limitations. We discuss the implications of these findings for the development of future AD tools for BLV users.

Figures (13)

• Figure 1: Figure (a) shows the test video used to familiarize participants with activating descriptions. Figures (b) through (h) illustrate the videos included in the study. The viewing order of the seven videos was randomized for each participant in the user study.
• Figure 2: Interface to test user-driven descriptions with (A) a video viewing section and video restart button, (B) a description box displaying the current timestamp, description, and a play button to replay the description, and (C) a button to proceed to the questionnaire and rate their experience of user-driven interaction with the video. When the user activates the description by pressing "C" for concise or "D" for detailed on the keypad, the video pauses to play the description and resumes once the description is over.
• Figure 3: Distribution of Participant Ratings Across Video Genres
• Figure 4: Results for the frequency and type of AD activations in the seven videos.
• Figure 5: Patterns of concise and detailed AD activations from example blind and low vision participants. On each plot, the rows show the seven videos, and the horizontal axis shows the video timeline in seconds. The video durations were between 52 and 107 seconds.