Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition

Dominik Wagner, Ilja Baumann, Natalie Engert, Seanie Lee, Elmar Nöth, Korbinian Riedhammer, Tobias Bocklet

TL;DR

This work tackles dysarthric speech recognition by combining speaker-adaptive personalization with parameter-efficient fine-tuning in a Whisper-based encoder-decoder, complemented by controllable synthetic data generation. By fine-tuning Parler-TTS to produce dysarthric-like speech driven by LLM-generated transcripts and injecting x-vectors into the decoder, the authors demonstrate substantial WER reductions (up to ~31% relative) compared to non-personalized baselines. They further show that incorporating wav2vec 2.0 audio representations adds ~5% additional WER improvement and that synthetic dysarthric data can yield up to ~7% extra gains, though effectiveness varies by etiology and TTS model. The approach generalizes beyond Whisper to HuBERT with additive benefits and presents a practical, data-efficient pathway for personalized, accessible dysarthric ASR, leveraging controllable generation of synthetic data and targeted adapter-based fine-tuning.

Abstract

In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ~23% and ~22%, respectively. Further improvements (~5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ~7% relative WER improvement over personalized fine-tuning alone.

Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition

TL;DR

This work tackles dysarthric speech recognition by combining speaker-adaptive personalization with parameter-efficient fine-tuning in a Whisper-based encoder-decoder, complemented by controllable synthetic data generation. By fine-tuning Parler-TTS to produce dysarthric-like speech driven by LLM-generated transcripts and injecting x-vectors into the decoder, the authors demonstrate substantial WER reductions (up to ~31% relative) compared to non-personalized baselines. They further show that incorporating wav2vec 2.0 audio representations adds ~5% additional WER improvement and that synthetic dysarthric data can yield up to ~7% extra gains, though effectiveness varies by etiology and TTS model. The approach generalizes beyond Whisper to HuBERT with additive benefits and presents a practical, data-efficient pathway for personalized, accessible dysarthric ASR, leveraging controllable generation of synthetic data and targeted adapter-based fine-tuning.

Abstract

In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ~23% and ~22%, respectively. Further improvements (~5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ~7% relative WER improvement over personalized fine-tuning alone.
Paper Structure (13 sections, 5 equations, 1 figure, 4 tables)

This paper contains 13 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Method
  4. Model adaptation
  5. Synthetic dysarthric speech generation
  6. Target prompt generation
  7. Experiments and results
  8. Main results
  9. Generalization beyond Whisper
  10. Adding additional audio representations
  11. Adding synthetic speech data
  12. Limitations
  13. Conclusions

Figures (1)

  • Figure 1: Overview of the proposed ASR system combining the incorporation of audio representations, fine-tuning, and synthetic training data generation.