Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Rehearsal for Continual Learning in ASR via Singular Value Tuning

Steven Vander Eeckt, Hugo Van hamme

TL;DR

This paper tackles catastrophic forgetting in continual learning for automatic speech recognition under strict memory constraints. It introduces Singular Value-based Rehearsal (SVR), a two-stage method that first fine-tunes on the new task and then applies an SVD of the linear weight changes, with a learnable gating vector on the singular values to selectively accept updates using rehearsal memory. SVR updates only a small set of gating parameters while freezing most of the model, achieving strong retention of past tasks and robust adaptation to new ones across mono- and multilingual ASR benchmarks, even with as little as a single utterance per task. The approach provides interpretable mechanisms via binary-like gating on rank-one updates and demonstrates substantial improvements over state-of-the-art rehearsal-based and regularization-based CL methods, with potential for broader applicability and extensions to memory selection and PEFT techniques.

Abstract

Continual Learning (CL) in Automatic Speech Recognition (ASR) suffers from catastrophic forgetting when adapting to new tasks, domains, or speakers. A common strategy to mitigate this is to store a subset of past data in memory for rehearsal. However, rehearsal-based methods face key limitations: storing data is often costly, infeasible with pre-trained models, or restricted by privacy regulations. Running existing rehearsal-based methods with smaller memory sizes to alleviate these issues usually leads to degraded performance. We propose a rehearsal-based CL method that remains effective even with minimal memory. It operates in two stages: first, fine-tuning on the new task; second, applying Singular Value Decomposition (SVD) to the changes in linear layers and, in a parameter-efficient manner, retraining only gating vectors on the singular values, which control to extent to which updates from the first stage are accepted, using rehearsal. We extensively test and analyze our method on two monolingual and two multilingual benchmarks. Our method reduces forgetting and outperforms state-of-the-art CL approaches for ASR, even when limited to a single utterance per previous task.

Efficient Rehearsal for Continual Learning in ASR via Singular Value Tuning

TL;DR

This paper tackles catastrophic forgetting in continual learning for automatic speech recognition under strict memory constraints. It introduces Singular Value-based Rehearsal (SVR), a two-stage method that first fine-tunes on the new task and then applies an SVD of the linear weight changes, with a learnable gating vector on the singular values to selectively accept updates using rehearsal memory. SVR updates only a small set of gating parameters while freezing most of the model, achieving strong retention of past tasks and robust adaptation to new ones across mono- and multilingual ASR benchmarks, even with as little as a single utterance per task. The approach provides interpretable mechanisms via binary-like gating on rank-one updates and demonstrates substantial improvements over state-of-the-art rehearsal-based and regularization-based CL methods, with potential for broader applicability and extensions to memory selection and PEFT techniques.

Abstract

Continual Learning (CL) in Automatic Speech Recognition (ASR) suffers from catastrophic forgetting when adapting to new tasks, domains, or speakers. A common strategy to mitigate this is to store a subset of past data in memory for rehearsal. However, rehearsal-based methods face key limitations: storing data is often costly, infeasible with pre-trained models, or restricted by privacy regulations. Running existing rehearsal-based methods with smaller memory sizes to alleviate these issues usually leads to degraded performance. We propose a rehearsal-based CL method that remains effective even with minimal memory. It operates in two stages: first, fine-tuning on the new task; second, applying Singular Value Decomposition (SVD) to the changes in linear layers and, in a parameter-efficient manner, retraining only gating vectors on the singular values, which control to extent to which updates from the first stage are accepted, using rehearsal. We extensively test and analyze our method on two monolingual and two multilingual benchmarks. Our method reduces forgetting and outperforms state-of-the-art CL approaches for ASR, even when limited to a single utterance per previous task.
Paper Structure (26 sections, 8 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 26 sections, 8 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Continual Learning for ASR
  3. Model and Notation
  4. Problem Formulation
  5. Related Work
  6. Singular Value-based Rehearsal
  7. Stage 1: Fine-Tuning to learn the new task $t$
  8. Stage 2a: Preparing Singular Value-Based Rehearsal
  9. Stage 2b: Singular Value-Based Rehearsal
  10. Summary
  11. Experiments
  12. Results
  13. Experiment 1
  14. Experiment 2
  15. Experiment 3
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Average WER improvement (%) over Fine-Tuning, reported per method and after each of the five tasks. The values indicate the relative percentage reduction in Average WER compared to Fine-Tuning at each task.
  • Figure 2: Comparison of performance of rehearsal-based baselines (KD and ER) and our method (SVR) for different memory sizes $|{\cal M}|$. On the left, the y-axis shows the Average WER, where lower is better. On the right, it shows Forgetting, equal to -BWT, such that, here too, lower is better.
  • Figure 3: Plot of $\sigma(\bm{\alpha})$ across layers (to which our method is applied) for SVR [$|{\cal M}|=1$] after the first adaptation (1--US $\rightarrow$ 2--ENG) of Exp. 1. The layers are sorted from input (left) to output (right), with the dotted line indicating the end of encoder and start of decoder. Layers are colored based on layer type. Fig. \ref{['fig:alphas_mean']} gives the mean $\sigma(\bm{\alpha})$ value per layer, while Fig. \ref{['fig:alphas_ratio']} gives the percentage of $\sigma(\bm{\alpha})$ which differ significantly (by more than 0.05) from 0 or 1, i.e. the percentage of $\sigma(\bm{\alpha})$ for which $0.05 < \sigma(\alpha_i)<0.95$.
  • Figure 4: Mean and percentage of intermediate and small values of $\sigma(\bm{\alpha})$ across layer groups after learning each task for SVR [$|{\cal M}|{=}1$] on Exp. 1. The layer groups are: (1) encoder key-query (k/q) attention; (2) encoder value-output (v/o) attention; (3) encoder feedforward; (4) output layers (CTC and decoder output); (5) decoder key-query (k/q) attention; (6) decoder value-output (v/o) attention; and (7) decoder feedforward. Fig. \ref{['fig:alphas_mean_over_time']} shows the mean $\sigma(\bm{\alpha})$ per group after learning each task. Fig. \ref{['fig:alphas_ratio_over_time']} shows the percentage of parameters in $\sigma(\bm{\alpha})$ for which $0.05 < \sigma(\alpha_i) < 0.95$. Fig. \ref{['fig:alphas_zeroratio_over_time']} shows the percentage for which $\sigma(\alpha_i) < 0.05$.