StutterZero and StutterFormer: End-to-End Speech Conversion for Stuttering Transcription and Correction

TL;DR

This paper tackles the challenge that stuttering disrupts transcription and communication by evaluating end-to-end waveform-to-waveform models. It introduces two multitask architectures, StutterZero and StutterFormer, that simultaneously restore fluency and predict transcription from stuttered speech. Trained on SEP-28K and LibriStutter and evaluated on FluencyBank, the end-to-end models significantly outperform strong fluent-speech baselines, achieving markedly lower WER and higher semantic similarity. The results support the feasibility and practicality of direct stutter-to-fluent speech conversion, with potential impact on speech therapy, inclusive human-computer interaction, and accessibility-focused AI. Limitations include reliance on TTS-generated data and dataset diversity, guiding future work toward prosody-aware tuning, multilingual expansion, clinical integration, and real-time deployment.

Abstract

Over 70 million people worldwide experience stuttering, yet most automatic speech systems misinterpret disfluent utterances or fail to transcribe them accurately. Existing methods for stutter correction rely on handcrafted feature extraction or multi-stage automatic speech recognition (ASR) and text-to-speech (TTS) pipelines, which separate transcription from audio reconstruction and often amplify distortions. This work introduces StutterZero and StutterFormer, the first end-to-end waveform-to-waveform models that directly convert stuttered speech into fluent speech while jointly predicting its transcription. StutterZero employs a convolutional-bidirectional LSTM encoder-decoder with attention, whereas StutterFormer integrates a dual-stream Transformer with shared acoustic-linguistic representations. Both architectures are trained on paired stuttered-fluent data synthesized from the SEP-28K and LibriStutter corpora and evaluated on unseen speakers from the FluencyBank dataset. Across all benchmarks, StutterZero had a 24% decrease in Word Error Rate (WER) and a 31% improvement in semantic similarity (BERTScore) compared to the leading Whisper-Medium model. StutterFormer achieved better results, with a 28% decrease in WER and a 34% improvement in BERTScore. The results validate the feasibility of direct end-to-end stutter-to-fluent speech conversion, offering new opportunities for inclusive human-computer interaction, speech therapy, and accessibility-oriented AI systems.

Figures (5)

• Figure 1: General pipeline of an ASR-TTS approach.
• Figure 2: Conversion from waveform to log-Mel spectrogram. The waveform (left) shows amplitude over time. After STFT, the spectrogram (middle) represents frequency over time. Applying the Mel filterbank produces a log-Mel spectrogram (right), aligned with human auditory perception.
• Figure 3: Overview of the StutterZero end-to-end architecture. A convolutional BiLSTM encoder compresses these features into a 512-dimensional context vector, which is shared between two multitask decoders. The spectrogram decoder predicts fluent spectrogram frames that are reconstructed into audio using the Griffin–Lim algorithm, while the transcript decoder predicts grapheme tokens.
• Figure 4: Overview of the StutterFormer end-to-end model. The encoder projects features into a 128-dimensional space, applies sinusoidal positional encoding, and processes them through stacked multi-head attention and feed-forward layers to produce a 256-dimensional context vector. This vector is shared between two multitask decoders. The spectrogram decoder refines fluent spectrograms via masked self-attention, cross attention, and convolutional post-nets, while the transcript decoder predicts grapheme tokens using masked self-attention and cross attention layers. The Griffin–Lim algorithm reconstructs waveforms from refined spectrograms, jointly optimizing fluency restoration and transcription.
• Figure 5: The validation pipeline. An untouched Whisper-Small ASR model acted as "judge" to evaluate speech clarity and intelligibility.