Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Si Chen, Haocong Cheng, Jason Situ, Desirée Kirst, Suzy Su, Saumya Malhotra, Lawrence Angrave, Qi Wang, Yun Huang

TL;DR

Signmaku introduces an ASL based sign language danmaku to make video based learning more inclusive for Deaf and hard of hearing students. Through a two phase study comparing Realistic, Cartoon, and Robotic signmaku styles (N=12 in formative rounds; N=20 in the evaluation), Cartoon ASL comments emerged as engaging while preserving privacy, Realistic ASL supported comprehension, and Robotic ASL imposed high cognitive load. The findings yield design implications for AI generated ASL content, privacy preserving filters, and scalable parameter tuning to support inclusive co learning. The work advances a new edu taintment and peer sourced interaction paradigm for DHH learners, with practical impact on accessibility in online education, while noting limitations of current AI sign language generation and privacy considerations.

Abstract

Previous research underscored the potential of danmaku--a text-based commenting feature on videos--in engaging hearing audiences. Yet, for many Deaf and hard-of-hearing (DHH) individuals, American Sign Language (ASL) takes precedence over English. To improve inclusivity, we introduce "Signmaku," a new commenting mechanism that uses ASL, serving as a sign language counterpart to danmaku. Through a need-finding study (N=12) and a within-subject experiment (N=20), we evaluated three design styles: real human faces, cartoon-like figures, and robotic representations. The results showed that cartoon-like signmaku not only entertained but also encouraged participants to create and share ASL comments, with fewer privacy concerns compared to the other designs. Conversely, the robotic representations faced challenges in accurately depicting hand movements and facial expressions, resulting in higher cognitive demands on users. Signmaku featuring real human faces elicited the lowest cognitive load and was the most comprehensible among all three types. Our findings offered novel design implications for leveraging generative AI to create signmaku comments, enriching co-learning experiences for DHH individuals.

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

TL;DR

Signmaku introduces an ASL based sign language danmaku to make video based learning more inclusive for Deaf and hard of hearing students. Through a two phase study comparing Realistic, Cartoon, and Robotic signmaku styles (N=12 in formative rounds; N=20 in the evaluation), Cartoon ASL comments emerged as engaging while preserving privacy, Realistic ASL supported comprehension, and Robotic ASL imposed high cognitive load. The findings yield design implications for AI generated ASL content, privacy preserving filters, and scalable parameter tuning to support inclusive co learning. The work advances a new edu taintment and peer sourced interaction paradigm for DHH learners, with practical impact on accessibility in online education, while noting limitations of current AI sign language generation and privacy considerations.

Abstract

Previous research underscored the potential of danmaku--a text-based commenting feature on videos--in engaging hearing audiences. Yet, for many Deaf and hard-of-hearing (DHH) individuals, American Sign Language (ASL) takes precedence over English. To improve inclusivity, we introduce "Signmaku," a new commenting mechanism that uses ASL, serving as a sign language counterpart to danmaku. Through a need-finding study (N=12) and a within-subject experiment (N=20), we evaluated three design styles: real human faces, cartoon-like figures, and robotic representations. The results showed that cartoon-like signmaku not only entertained but also encouraged participants to create and share ASL comments, with fewer privacy concerns compared to the other designs. Conversely, the robotic representations faced challenges in accurately depicting hand movements and facial expressions, resulting in higher cognitive demands on users. Signmaku featuring real human faces elicited the lowest cognitive load and was the most comprehensible among all three types. Our findings offered novel design implications for leveraging generative AI to create signmaku comments, enriching co-learning experiences for DHH individuals.
Paper Structure (39 sections, 3 figures, 1 table)

This paper contains 39 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges Faced by DHH Students in Video-based Learning
  4. Danmaku for Social Interaction and Knowledge Sharing in Video-based Learning
  5. Anonymization of Sign Language Videos for Privacy Concerns
  6. Method
  7. Phase I: Designing Signmaku--Video Comments in Sign Language
  8. Round 1: Needfinding of Signmaku for Social Connectedness
  9. Round 2: Specifying Design Features of Signmaku
  10. Phase II: Comparing Three Styled Signmaku
  11. Participants Information
  12. Experimental Design
  13. Measurements and Data Analysis
  14. Findings
  15. Engaging Learners through Viewing Three Styles of Signmaku (RQ1)
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Three Styles of Signmaku designs explored in own study: realistic (unfiltered), cartoon (filtered face with real torso and hands), and robotic (filtered face, torso, and hands) leverages AI technology to filter signed video clips, considering different styles of privacy preservation. Cartoon filter's background was selected to contrast an individual's skin color for more visible hand movements using VToonify zhang2022s. Robotic filter's appearance was customized by the DHH individual in DeepMotion.
  • Figure 2: Experiment Design. Each participant completed two activities during the study. First, they watched a video about augmented reality (AR) with error-free open captions and signmakus (RQ1). Second, they provided their comments in signmaku (ASL comment) or text comments (RQ2). A comment may be consisted entirely of text, solely use ASL, or, on very rare occasions, include a mix of both. They then completed a post-study survey and interviews. Participants were ask to provide signmaku only after they finished watching the video for the first time.
  • Figure 3: Boxplots of participants' reported mental demand, physical demand, and time pressure of viewing three styles of signmakus (N=20) on 10-point Likert scales (Low to High). The mental demand, physical demand, and time pressure were all significantly higher when viewing robotic ASL signmakus compared to the realistic styles. Note **, *** signify p< .01 and .001, respectively.