STEAMROLLER: A Multi-Agent System for Inclusive Automatic Speech Recognition for People who Stutter

TL;DR

STEAMROLLER tackles inclusive ASR for people who stutter by converting stuttered speech into fluent audio in real time. It uses a three-stage pipeline—ASR transcription, semantic-aware multi-agent text repair, and diffusion-based voice cloning for synthesis—operating with a cooperative multi-agent framework to preserve semantic intent. Empirical results on FluencyBank and a 23-participant user study show reductions in WER, MER, and WIL, improved semantic similarity, and high user satisfaction; fine-tuning ASR on repaired speech yields additional gains. The work demonstrates a practical pathway to inclusive voice interfaces and datasets for disordered speech, with potential benefits for accessibility, annotation, and model training.

Abstract

People who stutter (PWS) face systemic exclusion in today's voice-driven society, where access to voice assistants, authentication systems, and remote work tools increasingly depends on fluent speech. Current automatic speech recognition (ASR) systems, trained predominantly on fluent speech, fail to serve millions of PWS worldwide. We present STEAMROLLER, a real time system that transforms stuttered speech into fluent output through a novel multi-stage, multi-agent AI pipeline. Our approach addresses three critical technical challenges: (1) the difficulty of direct speech to speech conversion for disfluent input, (2) semantic distortions introduced during ASR transcription of stuttered speech, and (3) latency constraints for real time communication. STEAMROLLER employs a three stage architecture comprising ASR transcription, multi-agent text repair, and speech synthesis, where our core innovation lies in a collaborative multi-agent framework that iteratively refines transcripts while preserving semantic intent. Experiments on the FluencyBank dataset and a user study demonstrates clear word error rate (WER) reduction and strong user satisfaction. Beyond immediate accessibility benefits, fine tuning ASR on STEAMROLLER repaired speech further yields additional WER improvements, creating a pathway toward inclusive AI ecosystems.

Figures (3)

• Figure 1: Comparison of ASR transcription outcomes for stutter-free and stuttering audio inputs. For the same sentence, stuttering speech may lead to errors in recognizing words and ultimately transcribing incorrect text, highlighting the inaccuracies in text generation from stuttered speech.
• Figure 2: The process of stuttering speech to text conversion. In this example, the word "different" includes several repeated syllables due to stuttering, which ultimately causes it to be incorrectly recognized as multiple words.
• Figure 3: The workflow of Steamroller. First, an ASR system transcribes the stuttered audio into text. Next, an NLP processor converts it into syllable or token sequences. Then, a multiagent cluster addresses potential misunderstandings in the transcribed text, with multiple repair agents responsible for making corrections. A master agent subsequently evaluates them and provides the final Repaired text. Finally, voice cloning technology transforms corrected text into stutter-free audio.