Tailored Real-time AR Captioning Interface for Enhancing Learning Experience of Deaf and Hard-of-Hearing (DHH) Students

TL;DR

This work targets the language-literacy barriers faced by Deaf and hard-of-hearing students in specialized educational settings by introducing a real-time AR captioning interface developed through a rigorous user-centered design process. The system combines AR glasses with a Unity-based application and Azure Speech-to-Text to deliver localized Sinhala captions in the learner’s line of sight, while offering multiple caption presentation, placement, and markup options informed by direct input from DHH students and teachers. Across five user studies with 24 participants, the Karaoke-Like caption presentation and near-speaker placement consistently yielded the best learning outcomes, with significant gains in comprehension ($p<.001$) and reading speed, and strong overall usability ($SUS$ ≈ $78.64$). The study also reveals nuanced preferences for markup types and personal-utterance visualization, highlighting the need for adaptable, user-controllable AR captioning to support literacy development, engagement, and potential mainstream integration, while noting practical limitations and directions for longitudinal evaluation. Overall, the findings offer actionable guidance for designing accessible, AR-based educational tools tailored to DHH learners and set the stage for broader, long-term impact assessments in diverse classroom contexts.

Abstract

Deaf and hard-of-hearing (DHH) students face significant challenges in specialized educational settings, such as limited exposure to written and spoken language, a lack of tailored educational tools, and restricted access to resources, impacting their language literacy development and overall educational experience. We, therefore, employed a User-Centered Design (UCD) process, collaborating with 8 DHH students and 2 Teachers of the Deaf (ToDs) from a School of Deaf to effectively develop and utilize a real-time captioning augmented reality (AR) system to their school settings, aiming to enhance their learning experience. User study with 24 DHH participants revealed a strong preference (87.5\%) for our system, underscoring its potential to enhance learning experience. We present a comprehensive needs analysis, the UCD process, system implementation, and user feedback, showcasing the effectiveness of tailored AR caption interfaces for DHH students. We also discuss the implications for future development of educational technologies for DHH students.

Figures (16)

• Figure 1: Initial Prototype (Non-optimized AR interface - 4 lines of text with 20 words) [Refer the Appendix \ref{['appendix']} for English translations]
• Figure 2: User-Centered Design Process (Semi-Structured Interviews, Focus Groups, Co-Design Sessions)
• Figure 3: Designs envisioned by DHH Students: (a) Illustrating the number of caption lines (one or two), words per line (three), and the placement (left to the speaker, right to the speaker, directly below), (b) Markups for inaccurately transcribed word (a-c), (c) Illustrating using different colors to separate personal utterances [Refer the Appendix \ref{['appendix']} for English translations]
• Figure 4: Caption presentation methods [Refer the Appendix \ref{['appendix']} for English translations]
• Figure 5: Text highlighting strategies [Refer the Appendix \ref{['appendix']} for English translations]