Text-only adaptation in LLM-based ASR through text denoising

TL;DR

The paper addresses adapting LLM-based ASR to new domains using text-only data by reframing adaptation as a text denoising problem that mimics the projector's outputs. It introduces a lightweight, architecture-free batch-mixing strategy that preserves speech–text alignment while enabling target-domain specialization, controlled by an adaptation weight $\\tau$. Empirical results on DefinedAI and SlideSpeech show consistent WER improvements, often approaching audio-based adaptation and surpassing prior text-only methods, with up to 22.1% relative gains reported. This approach enables scalable domain adaptation without collecting new audio data, leveraging existing text resources effectively.

Abstract

Adapting automatic speech recognition (ASR) systems based on large language models (LLMs) to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on target-domain text often disrupts the critical alignment between speech and text modalities learned by the projector, degrading performance. We introduce a novel text-only adaptation method that emulates the audio projection task by treating it as a text denoising task. Our approach thus trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.

Figures (1)

• Figure 1: Batch composition used for fine-tuning the LLM during text-only adaptation to a target domain. Here, $\sigma$ and $\boldsymbol\tau$ denote the proportions of the batch that are drawn from the source domain ($\mathcal{D}_{src}$) and the target domain ($\mathcal{D}_{tgt}$), respectively.