DanmuA11y: Making Time-Synced On-Screen Video Comments (Danmu) Accessible to Blind and Low Vision Users via Multi-Viewer Audio Discussions

TL;DR

DanmuA11y tackles the accessibility of time-synced Danmu comments for blind and low-vision viewers by converting them into multi-viewer audio discussions. It introduces three core features—augmenting Danmu with visual context, seamless video integration to avoid speech overlap, and multi-viewer audio presentations—to create a co-watching experience. The approach combines GPT-4o-based topic modeling for grouping, automated visual description generation, an ILP-based insertion optimization, and spatial audio rendering, all implemented in an iOS app. Evaluation with 12 BLV participants shows significant improvements in Danmu comprehension, viewing smoothness, and social connectedness, suggesting strong potential for broader commentary accessibility in video and live-streaming platforms. The work offers design implications and practical pathways for enhancing accessibility in time-synced visual content.

Abstract

By overlaying time-synced user comments on videos, Danmu creates a co-watching experience for online viewers. However, its visual-centric design poses significant challenges for blind and low vision (BLV) viewers. Our formative study identified three primary challenges that hinder BLV viewers' engagement with Danmu: the lack of visual context, the speech interference between comments and videos, and the disorganization of comments. To address these challenges, we present DanmuA11y, a system that makes Danmu accessible by transforming it into multi-viewer audio discussions. DanmuA11y incorporates three core features: (1) Augmenting Danmu with visual context, (2) Seamlessly integrating Danmu into videos, and (3) Presenting Danmu via multi-viewer discussions. Evaluation with twelve BLV viewers demonstrated that DanmuA11y significantly improved Danmu comprehension, provided smooth viewing experiences, and fostered social connections among viewers. We further highlight implications for enhancing commentary accessibility in video-based social media and live-streaming platforms.

Figures (11)

• Figure 1: The key characteristics of Danmu comments.
• Figure 2: The design probe presents Danmu comments in an auto-scrolling list. Participants used the probe in Mandarin.
• Figure 3: An example walk-through of DanmuA11y: (A) The system automatically plays multi-viewer discussions during non-speech segments, using spatial audio and varied tones to create a co-watching experience. (B) The system provides on-demand access at speech breaks, notifying users with a sound and allowing them to access Danmu comments by shaking their phones.
• Figure 4: The pipeline of DanmuA11y.
• Figure 5: DanmuA11y divides the video into non-speech and speech segments. The points between adjacent speech segments are called speech breaks.