Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mind the Shift: Using Delta SSL Embeddings to Enhance Child ASR

Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Zihan Wang, Abeer Alwan

TL;DR

This paper addresses the challenge of child ASR by leveraging delta SSL embeddings, defined as the difference between fine-tuned and pre-trained embeddings, to fuse diverse SSL representations. The proposed approach combines delta embeddings with fine-tuned embeddings from another SSL model using simple concatenation, outperforming other fusion methods and achieving a new state-of-the-art WER of 9.64 on the MyST corpus among SSL models. Analyses using Canonical Correlation Analysis and a Mixture-of-Experts gate reveal that delta embeddings capture task-specific shifts concentrated in upper layers and provide complementary information, including across domain transfers. The findings demonstrate that a straightforward fusion of delta and fine-tuned features can substantially enhance child ASR, particularly in extremely low-resource settings, with potential applicability to other low-resource domains.

Abstract

Self-supervised learning (SSL) models have achieved impressive results across many speech tasks, yet child automatic speech recognition (ASR) remains challenging due to limited data and pretraining domain mismatch. Fine-tuning SSL models on child speech induces shifts in the representation space. We hypothesize that delta SSL embeddings, defined as the differences between embeddings from a finetuned model and those from its pretrained counterpart, encode task-specific information that complements finetuned features from another SSL model. We evaluate multiple fusion strategies on the MyST childrens corpus using different models. Results show that delta embedding fusion with WavLM yields up to a 10 percent relative WER reduction for HuBERT and a 4.4 percent reduction for W2V2, compared to finetuned embedding fusion. Notably, fusing WavLM with delta W2V2 embeddings achieves a WER of 9.64, setting a new state of the art among SSL models on the MyST corpus. These findings demonstrate the effectiveness of delta embeddings and highlight feature fusion as a promising direction for advancing child ASR.

Mind the Shift: Using Delta SSL Embeddings to Enhance Child ASR

TL;DR

This paper addresses the challenge of child ASR by leveraging delta SSL embeddings, defined as the difference between fine-tuned and pre-trained embeddings, to fuse diverse SSL representations. The proposed approach combines delta embeddings with fine-tuned embeddings from another SSL model using simple concatenation, outperforming other fusion methods and achieving a new state-of-the-art WER of 9.64 on the MyST corpus among SSL models. Analyses using Canonical Correlation Analysis and a Mixture-of-Experts gate reveal that delta embeddings capture task-specific shifts concentrated in upper layers and provide complementary information, including across domain transfers. The findings demonstrate that a straightforward fusion of delta and fine-tuned features can substantially enhance child ASR, particularly in extremely low-resource settings, with potential applicability to other low-resource domains.

Abstract

Self-supervised learning (SSL) models have achieved impressive results across many speech tasks, yet child automatic speech recognition (ASR) remains challenging due to limited data and pretraining domain mismatch. Fine-tuning SSL models on child speech induces shifts in the representation space. We hypothesize that delta SSL embeddings, defined as the differences between embeddings from a finetuned model and those from its pretrained counterpart, encode task-specific information that complements finetuned features from another SSL model. We evaluate multiple fusion strategies on the MyST childrens corpus using different models. Results show that delta embedding fusion with WavLM yields up to a 10 percent relative WER reduction for HuBERT and a 4.4 percent reduction for W2V2, compared to finetuned embedding fusion. Notably, fusing WavLM with delta W2V2 embeddings achieves a WER of 9.64, setting a new state of the art among SSL models on the MyST corpus. These findings demonstrate the effectiveness of delta embeddings and highlight feature fusion as a promising direction for advancing child ASR.
Paper Structure (21 sections, 4 equations, 2 figures, 5 tables)

This paper contains 21 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Feature Extraction
  4. Feature Fusion Techniques
  5. Canonical Correlation Analysis
  6. Mixture of Experts
  7. Experimental Setup
  8. Datasets
  9. Models
  10. Feature Fusion
  11. Results
  12. Do delta embeddings improve fusion performance?
  13. Comparison of Fusion Methods
  14. Comparison with Individual Fine-Tuned Models
  15. Fusion of Fine-Tuned vs. Delta Embeddings
  16. ...and 6 more sections

Figures (2)

  • Figure 1: CCA similarity between pre-trained and fine-tuned models.
  • Figure 2: CCA similarity between fine-tuned and $\Delta$ embeddings.