Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks

Amit Meghanani, Thomas Hain

TL;DR

LASER tackles the efficiency gap in improving content-related speech tasks using self-supervised representations. It introduces a cost-effective SSFT method that aligns content-bearing embeddings from original and perturbed speech via a soft-DTW loss augmented with temporal regularisation to prevent representation collapse. The approach improves HuBERT and WavLM on ASR and phoneme recognition in the SUPERB benchmark with under 3 hours of fine-tuning on a single GPU, outperforming SCORE and competing with SPIN while using far less compute than ContentVec. The key insight is that temporal regularisation is essential to preserve content information during alignment, enabling robust, content-focused fine-tuning. Overall, LASER offers a practical, scalable path to leveraging SSL representations for content tasks in resource-constrained settings.

Abstract

Self-supervised learning (SSL)-based speech models are extensively used for full-stack speech processing. However, it has been observed that improving SSL-based speech representations using unlabeled speech for content-related tasks is challenging and computationally expensive. Recent attempts have been made to address this issue with cost-effective self-supervised fine-tuning (SSFT) approaches. Continuing in this direction, a cost-effective SSFT method named "LASER: Learning by Aligning Self-supervised Representations" is presented. LASER is based on the soft-DTW alignment loss with temporal regularisation term. Experiments are conducted with HuBERT and WavLM models and evaluated on the SUPERB benchmark for two content-related tasks: automatic speech recognition (ASR) and phoneme recognition (PR). A relative improvement of 3.7% and 8.2% for HuBERT, and 4.1% and 11.7% for WavLM are observed, for the ASR and PR tasks respectively, with only < 3 hours of fine-tuning on a single GPU.

LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks

TL;DR

LASER tackles the efficiency gap in improving content-related speech tasks using self-supervised representations. It introduces a cost-effective SSFT method that aligns content-bearing embeddings from original and perturbed speech via a soft-DTW loss augmented with temporal regularisation to prevent representation collapse. The approach improves HuBERT and WavLM on ASR and phoneme recognition in the SUPERB benchmark with under 3 hours of fine-tuning on a single GPU, outperforming SCORE and competing with SPIN while using far less compute than ContentVec. The key insight is that temporal regularisation is essential to preserve content information during alignment, enabling robust, content-focused fine-tuning. Overall, LASER offers a practical, scalable path to leveraging SSL representations for content tasks in resource-constrained settings.

Abstract

Self-supervised learning (SSL)-based speech models are extensively used for full-stack speech processing. However, it has been observed that improving SSL-based speech representations using unlabeled speech for content-related tasks is challenging and computationally expensive. Recent attempts have been made to address this issue with cost-effective self-supervised fine-tuning (SSFT) approaches. Continuing in this direction, a cost-effective SSFT method named "LASER: Learning by Aligning Self-supervised Representations" is presented. LASER is based on the soft-DTW alignment loss with temporal regularisation term. Experiments are conducted with HuBERT and WavLM models and evaluated on the SUPERB benchmark for two content-related tasks: automatic speech recognition (ASR) and phoneme recognition (PR). A relative improvement of 3.7% and 8.2% for HuBERT, and 4.1% and 11.7% for WavLM are observed, for the ASR and PR tasks respectively, with only < 3 hours of fine-tuning on a single GPU.
Paper Structure (13 sections, 3 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 13 sections, 3 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experimental Setup
  4. Dataset and speech perturbations
  5. Details of model ($M_\theta$)
  6. Details of model head ($F_\mu$)
  7. Loss function
  8. LASER fine-tuning
  9. Evaluation on the SUPERB benchmark
  10. Results and Discussions
  11. Analysing the impact of regularisation
  12. Conclusions
  13. Acknowledgements

Figures (1)

  • Figure 1: LASER fine-tuning approach. The loss function is computed for the representations obtained from original speech ($X$) and perturbed speech ($X'$): $L(X,X')=\text{soft-DTW}_\gamma(X,X') + \alpha(f(X) + f(X'))$.