Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data

Wen Ding, Fan Qian

Abstract

Although state-of-the-art Speech Foundational Models can produce high-quality text pseudo-labels, applying Semi-Supervised Learning (SSL) for in-the-wild real-world data remains challenging due to its richer and more complex acoustics compared to curated datasets. To address the challenges, we introduce LESS (Large Language Model Enhanced Semi-supervised Learning), a versatile framework that uses Large Language Models (LLMs) to correct pseudo-labels generated on in-the-wild data. In the LESS framework, pseudo-labeled text from Automatic Speech Recognition (ASR) or Automatic Speech Translation (AST) of the unsupervised data is refined by an LLM, and further improved by a data filtering strategy. Across Mandarin ASR and Spanish-to-English AST evaluations, LESS delivers consistent gains, with an absolute Word Error Rate reduction of 3.8% on WenetSpeech, and BLEU score increase of 0.8 and 0.7, achieving 34.0 on Callhome and 64.7 on Fisher testsets respectively. These results highlight LESS's effectiveness across diverse languages, tasks, and domains. We have released the recipe as open source to facilitate further research in this area.

LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data

Abstract

Although state-of-the-art Speech Foundational Models can produce high-quality text pseudo-labels, applying Semi-Supervised Learning (SSL) for in-the-wild real-world data remains challenging due to its richer and more complex acoustics compared to curated datasets. To address the challenges, we introduce LESS (Large Language Model Enhanced Semi-supervised Learning), a versatile framework that uses Large Language Models (LLMs) to correct pseudo-labels generated on in-the-wild data. In the LESS framework, pseudo-labeled text from Automatic Speech Recognition (ASR) or Automatic Speech Translation (AST) of the unsupervised data is refined by an LLM, and further improved by a data filtering strategy. Across Mandarin ASR and Spanish-to-English AST evaluations, LESS delivers consistent gains, with an absolute Word Error Rate reduction of 3.8% on WenetSpeech, and BLEU score increase of 0.8 and 0.7, achieving 34.0 on Callhome and 64.7 on Fisher testsets respectively. These results highlight LESS's effectiveness across diverse languages, tasks, and domains. We have released the recipe as open source to facilitate further research in this area.

Paper Structure

This paper contains 14 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. LESS pipeline
  4. In-the-wild data collection and processing
  5. Experiments
  6. Curated supervised data
  7. Evaluation data
  8. Experimental setup
  9. Results and Analysis
  10. ZH ASR
  11. Results
  12. Discussions
  13. ES-to-EN AST
  14. Conclusions

Figures (1)

  • Figure 1: The proposed LESS pipeline is illustrated taking the ES-to-EN AST task as an example. The process begins by finetuning the SFM at T=0 with supervised Spanish data (orange arrow). Unsupervised Spanish data collected from real-world sources are segmented by a VAD module. These Spanish segments are then transcribed into English using the initial SFM (SFM 0). The resulting English transcriptions are further refined by an LLM. After this, a data filtering step is applied. The filtered, LLM-corrected transcriptions are combined with the original supervised data to further finetune the SFM. This iterative process continues until convergence.