Augmenting Captions with Emotional Cues: An AR Interface for Real-Time Accessible Communication
Sunday David Ubur
TL;DR
The paper tackles the challenge of making STEM education accessible to Deaf and Hard of Hearing learners by enriching real-time captions with emotional and multimodal cues within an AR headset. It introduces an AR-based captioning system built in Unity that fuses ASR with emotion and gesture sensing to annotate captions with tonal and gestural context. The study provides a design-driven evaluation showing reduced cognitive load and improved comprehension compared to baseline captions, and discusses personalization and readability trade-offs. The work demonstrates the potential of immersive, emotion-aware captioning to enhance understanding and engagement in STEM classrooms and points to a path for adaptive, multimodal accessibility tools.
Abstract
This paper introduces an augmented reality (AR) captioning framework designed to support Deaf and Hard of Hearing (DHH) learners in STEM classrooms by integrating non-verbal emotional cues into live transcriptions. Unlike conventional captioning systems that offer only plain text, our system fuses real-time speech recognition with affective and visual signal interpretation, including facial movements, gestures, and vocal tone, to produce emotionally enriched captions. These enhanced captions are rendered in an AR interface developed with Unity and provide contextual annotations such as speaker tone markers (e.g., "concerned") and gesture indicators (e.g., "nods"). The system leverages live camera and microphone input, processed through AI models to detect multimodal cues. Findings from preliminary evaluations suggest that this AR-based captioning approach significantly enhances comprehension and reduces cognitive effort compared to standard captions. Our work emphasizes the potential of immersive environments for inclusive, emotion-aware educational accessibility.