VideoA11y: Method and Dataset for Accessible Video Description

TL;DR

VideoA11y tackles the gap in accessible video descriptions for blind and low-vision viewers by combining multimodal large language models with professional audio description guidelines. The authors introduce VideoA11y and VideoA11y-40K, the largest BLV-tailored video-description dataset, and validate their approach through five user studies and extensive benchmarks. Results show that MLLM-generated descriptions can surpass novice human annotations and rival trained humans in descriptiveness, objectivity, accuracy, and clarity, with strong satisfaction among BLV participants. The work also demonstrates that fine-tuning open-source MLLMs on VideoA11y-40K yields significant gains on both standard and four custom BLV-oriented metrics, offering a scalable path toward widespread, cost-effective video accessibility.

Abstract

Video descriptions are crucial for blind and low vision (BLV) users to access visual content. However, current artificial intelligence models for generating descriptions often fall short due to limitations in the quality of human annotations within training datasets, resulting in descriptions that do not fully meet BLV users' needs. To address this gap, we introduce VideoA11y, an approach that leverages multimodal large language models (MLLMs) and video accessibility guidelines to generate descriptions tailored for BLV individuals. Using this method, we have curated VideoA11y-40K, the largest and most comprehensive dataset of 40,000 videos described for BLV users. Rigorous experiments across 15 video categories, involving 347 sighted participants, 40 BLV participants, and seven professional describers, showed that VideoA11y descriptions outperform novice human annotations and are comparable to trained human annotations in clarity, accuracy, objectivity, descriptiveness, and user satisfaction. We evaluated models on VideoA11y-40K using both standard and custom metrics, demonstrating that MLLMs fine-tuned on this dataset produce high-quality accessible descriptions. Code and dataset are available at https://people-robots.github.io/VideoA11y.

Figures (15)

• Figure 1: The human annotations and descriptions generated by VideoA11y for six consecutive frames of two sample video clips. Red underline indicates the errors in human annotations, green bold indicates the corrected facts, and blue italics indicates additional details.
• Figure 2: Overview of the VideoA11y pipeline. First, keyframes are extracted from the input video. Then, the keyframes, the prompt, AD guidelines, and optional human annotations are provided to MLLM, which generates accessible video descriptions.
• Figure 3: Results of Study 1 with 150 sighted MTurk users. VideoA11y (GPT) outperforms other methods on all metrics ($p<0.05$), followed by VideoA11y (GPT) w/o HA. HA: Human Annotation.
• Figure 4: Overview of video categories and description lengths in VideoA11y-40K.
• Figure 5: Results of Study 2 with 150 sighted MTurk users. VideoA11y outperforms other methods in all metrics ($p<0.001$), followed by VideoA11y w/o HA. HA: Human Annotation.