Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Captioning with Emotional Cues: Supporting DHH and Neurodivergent Learners in STEM

Sunday David Ubur, Eugenia Ha Rim Rho, Denis Gracanin

Abstract

Real-time captioning is vital for Deaf and Hard of Hearing (DHH) and neurodivergent learners (e.g., those with ADHD), yet it often omits emotional and non-verbal cues essential for comprehension. This omission is particularly consequential in STEM education, where cognitively demanding material can exacerbate the challenges faced by caption users across diverse ability profiles. In this paper, we present a design-oriented exploration of four captioning prototypes that embed emotional and multimodal cues, including facial expressions, body gestures, keyword highlighting, and emoji. Across a pilot and a main study with 24 participants, we found that certain prototypes reduced self-reported cognitive load and improved comprehension scores compared to traditional captions. Qualitative feedback reveals the importance of customizable caption features to accommodate neurodivergent users' preferences (e.g., ADHD or different levels of comfort with emojis). Our findings contribute to ongoing conversations in accessible technology research about how best to integrate emotional cues into captions in a way that is both usable and beneficial for a wide range of learners.

Adaptive Captioning with Emotional Cues: Supporting DHH and Neurodivergent Learners in STEM

Abstract

Real-time captioning is vital for Deaf and Hard of Hearing (DHH) and neurodivergent learners (e.g., those with ADHD), yet it often omits emotional and non-verbal cues essential for comprehension. This omission is particularly consequential in STEM education, where cognitively demanding material can exacerbate the challenges faced by caption users across diverse ability profiles. In this paper, we present a design-oriented exploration of four captioning prototypes that embed emotional and multimodal cues, including facial expressions, body gestures, keyword highlighting, and emoji. Across a pilot and a main study with 24 participants, we found that certain prototypes reduced self-reported cognitive load and improved comprehension scores compared to traditional captions. Qualitative feedback reveals the importance of customizable caption features to accommodate neurodivergent users' preferences (e.g., ADHD or different levels of comfort with emojis). Our findings contribute to ongoing conversations in accessible technology research about how best to integrate emotional cues into captions in a way that is both usable and beneficial for a wide range of learners.
Paper Structure (50 sections, 1 equation, 4 figures, 2 tables)

This paper contains 50 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Emotional Cues in Education
  5. Emotional Expressiveness in Real-Time Captions
  6. Integrating Multiple Modalities in Real-Time Captions
  7. Formative Pilot Study
  8. Procedure and Materials
  9. Pilot Feedback and Refinements
  10. Main Study
  11. Overview and Rationale
  12. Prototype Design
  13. Participants
  14. Accessibility Characteristics
  15. Rationale for Sample Composition
  16. ...and 35 more sections

Figures (4)

  • Figure 1: Proposed caption design mock-up
  • Figure 2: NASA-TLX scores for Prototypes 1--4.
  • Figure 3: Effectiveness of emotional expression components and readability across Prototypes 1--4.
  • Figure 4: Comparison of caption usage frequency, perceived usefulness, prototype preferences, and feedback on accessibility features across Deaf, Hard of Hearing, and neurodivergent respondents.