Inverse-Hessian Regularization for Continual Learning in ASR

TL;DR

This work tackles catastrophic forgetting in continual learning for automatic speech recognition by introducing Inverse Hessian Regularization (IHR), a memory-free merging step that leverages curvature information through a layerwise Kronecker-factored inverse Hessian. After fine-tuning on a new task, IHR adjusts the task-specific update by multiplying it with the inverse Hessian estimate of the previous task(s), effectively steering learning along directions that minimally impact past performance. The method remains lightweight by applying the inverse-Hessian correction only once per task and storing the last task's Hessian, while constraining the regularization to linear layers. Empirical results on two domain-adaptation CL benchmarks show that IHR consistently outperforms strong memory-free baselines with significantly reduced forgetting and improved adaptability, supported by ablations and analyses confirming the role of curvature information in directing updates.

Abstract

Catastrophic forgetting remains a major challenge for continual learning (CL) in automatic speech recognition (ASR), where models must adapt to new domains without losing performance on previously learned conditions. Several CL methods have been proposed for ASR, and, recently, weight averaging - where models are averaged in a merging step after fine-tuning - has proven effective as a simple memory-free strategy. However, it is heuristic in nature and ignores the underlying loss landscapes of the tasks, hindering adaptability. In this work, we propose Inverse Hessian Regularization (IHR), a memory-free approach for CL in ASR that incorporates curvature information into the merging step. After fine-tuning on a new task, the adaptation is adjusted through a Kronecker-factored inverse Hessian approximation of the previous task, ensuring that the model moves primarily in directions less harmful to past performance, while keeping the method lightweight. We evaluate IHR on two CL benchmarks and show that it significantly outperforms state-of-the-art baselines, reducing forgetting while improving adaptability. Ablation studies and analyses further confirm its effectiveness.

Figures (2)

• Figure 1: Illustration of catastrophic forgetting. Fine-tuning moves the model from $\bm \theta^{t-1}$ to $\tilde{\bm \theta}^t$, entering the low-loss region of the new task $t$ (orange) but leaving that of tasks $1,\dots,t{-}1$ (blue). By multiplying the adaptation $\tilde{\bm \theta}^t - \bm \theta^{t-1}$ with the inverse Hessian, resulting parameters $\bm \theta^{t}$ remain within the intersection of both low-loss regions.
• Figure 2: WER as a function of the hyper-parameter $\tau$. Colors indicate the reported task (1--US, 2--ENG, or Average WER), while line style and marker indicate whether inverse Hessian Regularization (IHR) is applied (solid, circles) or not (dashed, squares).