Real-Time Word-Level Temporal Segmentation in Streaming Speech Recognition

TL;DR

This work tackles the gap in real-time rich-text captioning where word-level text attributes cannot be dynamically altered to convey speaker intent. It introduces a real-time word-level temporal segmentation pipeline that aligns streaming ASR results with per-word timestamps and applies dynamic text decorations, demonstrated in a Unity-based prototype using Azure ASR. The approach enables more expressive captions by reflecting paralinguistic cues such as loudness, with potential impact for DHH users, language learners, and ASD communication supports. Future work focuses on reducing latency, optimizing the segmentation integration, and expanding language coverage and practical applications, including real-time translation and educational tools.

Abstract

Rich-text captions are essential to help communication for Deaf and hard-of-hearing (DHH) people, second-language learners, and those with autism spectrum disorder (ASD). They also preserve nuances when converting speech to text, enhancing the realism of presentation scripts and conversation or speech logs. However, current real-time captioning systems lack the capability to alter text attributes (ex. capitalization, sizes, and fonts) at the word level, hindering the accurate conveyance of speaker intent that is expressed in the tones or intonations of the speech. For example, ''YOU should do this'' tends to be considered as indicating ''You'' as the focus of the sentence, whereas ''You should do THIS'' tends to be ''This'' as the focus. This paper proposes a solution that changes the text decorations at the word level in real time. As a prototype, we developed an application that adjusts word size based on the loudness of each spoken word. Feedback from users implies that this system helped to convey the speaker's intent, offering a more engaging and accessible captioning experience.