Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment

TL;DR

This work tackles multi-stuttered speech classification by leveraging Whisper's encoder in a multi-label setup, addressing a gap where real-world speech often contains multiple disfluencies. The authors curate a high-quality multi-stuttered dataset by merging SEP-28k and FluencyBank and constructing concatenated samples to simulate multiple disfluencies, evaluated on FluencyBank as an external test. They demonstrate that freezing selective encoder layers can reduce trainable parameters from $20.27$ million to $3.29$ million while maintaining strong F1-scores (micro $0.88$, macro $0.85$, weighted $0.87$), with the last encoder layer identified as particularly informative for disfluency detection. The results indicate Whisper's superiority over Wav2Vec2.0 in this task and highlight a practical, parameter-efficient approach suitable for real-world, multilingual assessment workflows for speech-language pathology.

Abstract

The automated classification of stuttered speech has significant implications for timely assessments providing assistance to speech language pathologists. Despite notable advancements in the field, the cases in which multiple disfluencies occur in speech require attention. We have taken a progressive approach to fill this gap by classifying multi-stuttered speech more efficiently. The problem has been addressed by firstly curating a dataset of multi-stuttered disfluencies from open source dataset SEP-28k audio clips. Secondly, employing Whisper, a state-of-the-art speech recognition model has been leveraged by using its encoder and taking the problem as multi label classification. Thirdly, using a 6 encoder layer Whisper and experimenting with various layer freezing strategies, a computationally efficient configuration of the model was identified. The proposed configuration achieved micro, macro, and weighted F1-scores of 0.88, 0.85, and 0.87, correspondingly on an external test dataset i.e. Fluency-Bank. In addition, through layer freezing strategies, we were able to achieve the aforementioned results by fine-tuning a single encoder layer, consequently, reducing the model's trainable parameters from 20.27 million to 3.29 million. This research study unveils the contribution of the last encoder layer in the identification of disfluencies in stuttered speech. Consequently, it has led to a computationally efficient approach, 83.7% less parameters to train, making the proposed approach more adaptable for various dialects and languages.

Figures (4)

• Figure 1: An Illustrative Overview of Different Types of Disfluency in Stuttered Speech
• Figure 2: Pipeline for Curation of Multi-stuttered data: Condition 1 involves acquiring audio from the same episodes, Condition 2 requires audios featuring the same speakers, and Condition 3, proceeds with concatenation if two audios have distinct disfluency labels or both have the label "NoStutteredWords".
• Figure 3: Whisper Architecture Configuration Designed for Enhanced Classification of Multi-Stuttered Speech Patterns
• Figure 4: Comparative Performance Analysis Showing Superior Results of Whisper Over Wav2vec2.0 with Fewer Trainable Parameters