Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models for Dysfluency Detection in Stuttered Speech

Dominik Wagner, Sebastian P. Bayerl, Ilja Baumann, Korbinian Riedhammer, Elmar Nöth, Tobias Bocklet

TL;DR

This work tackles multi-label dysfluency detection in stuttered speech by casting the task as a language modeling problem that jointly leverages acoustic representations from wav2vec 2.0 and lexical inputs from ASR or phonetic transcriptions, all processed by a frozen Llama 2 backbone fine-tuned with LoRA. The approach demonstrates that domain-adapted acoustic features, particularly from the last wav2vec layer, combined with 1-best lexical inputs, yield competitive or superior F1 scores across English and German datasets, with notable gains for block dysfluencies. Across experiments, additional lexical variants (n-best, MBR) provide limited improvements, while word repetitions remain challenging, partially due to pretraining biases in the lexical models. Overall, the study shows that integrating non-lexical and lexical information via an LLM is a promising path toward more inclusive and effective dysfluency-aware speech technologies.

Abstract

Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components and support the development of more inclusive speech and language technologies. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, such as audio and video, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech. The experimental results show that our system effectively combines acoustic and lexical information and achieves competitive results on the multi-label stuttering detection task.

Large Language Models for Dysfluency Detection in Stuttered Speech

TL;DR

This work tackles multi-label dysfluency detection in stuttered speech by casting the task as a language modeling problem that jointly leverages acoustic representations from wav2vec 2.0 and lexical inputs from ASR or phonetic transcriptions, all processed by a frozen Llama 2 backbone fine-tuned with LoRA. The approach demonstrates that domain-adapted acoustic features, particularly from the last wav2vec layer, combined with 1-best lexical inputs, yield competitive or superior F1 scores across English and German datasets, with notable gains for block dysfluencies. Across experiments, additional lexical variants (n-best, MBR) provide limited improvements, while word repetitions remain challenging, partially due to pretraining biases in the lexical models. Overall, the study shows that integrating non-lexical and lexical information via an LLM is a promising path toward more inclusive and effective dysfluency-aware speech technologies.

Abstract

Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components and support the development of more inclusive speech and language technologies. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, such as audio and video, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech. The experimental results show that our system effectively combines acoustic and lexical information and achieves competitive results on the multi-label stuttering detection task.
Paper Structure (15 sections, 5 equations, 1 figure, 1 table)

This paper contains 15 sections, 5 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Data
  3. Method
  4. wav2vec 2.0
  5. Whisper
  6. Low-rank Adaptation
  7. Minimum Bayes Risk Decoding
  8. Phonetic Transcriptions
  9. Our Approach
  10. Modeling and Architecture Details
  11. Experiments
  12. Experimental Setup
  13. Results and Discussion
  14. Conclusions
  15. Acknowledgments

Figures (1)

  • Figure 1: Overview of the components in our dysfluency detection system. The left segments illustrate the extraction and processing of acoustic features. The right segments demonstrate the retrieval and processing of lexical features, as well as the feature combination within the LLM backbone.