Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SALSA: Speedy ASR-LLM Synchronous Aggregation

Ashish Mittal, Darshan Prabhu, Sunita Sarawagi, Preethi Jyothi

TL;DR

This paper introduces SALSA, a lightweight method to fuse a pretrained ASR decoder with a pretrained LLM decoder by learning a small set of projection adapters that map ASR decoder states into selected LLM layers. It preserves the ASR and LLM backbones, handles tokenization mismatches via cascading tokenization, and enforces synchronous decoding so the two decoders advance together in one pass. On eight low-resource languages from the FLEURS benchmark, SALSA delivers substantial WER improvements over baselines, including a 38% relative reduction in the best cases, and significantly faster training than prior LLM-ASR fusion approaches. The approach is parameter-efficient, scalable across languages, and demonstrates the practical potential of leveraging strong LLMs to boost low-resource ASR without heavy retraining.

Abstract

Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%.

SALSA: Speedy ASR-LLM Synchronous Aggregation

TL;DR

This paper introduces SALSA, a lightweight method to fuse a pretrained ASR decoder with a pretrained LLM decoder by learning a small set of projection adapters that map ASR decoder states into selected LLM layers. It preserves the ASR and LLM backbones, handles tokenization mismatches via cascading tokenization, and enforces synchronous decoding so the two decoders advance together in one pass. On eight low-resource languages from the FLEURS benchmark, SALSA delivers substantial WER improvements over baselines, including a 38% relative reduction in the best cases, and significantly faster training than prior LLM-ASR fusion approaches. The approach is parameter-efficient, scalable across languages, and demonstrates the practical potential of leveraging strong LLMs to boost low-resource ASR without heavy retraining.

Abstract

Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%.
Paper Structure (6 sections, 1 equation, 2 figures, 3 tables, 1 algorithm)

This paper contains 6 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Approach: SALSA
  4. Experimental Setup
  5. Experimental Results and Analysis
  6. Conclusion

Figures (2)

  • Figure 1: SALSA: Overall schematic illustrating the coupling of the ASR model ${\hbox{$\mathcal{M}$}}$ and the LLM $\mathcal{L}$ using select projection layers. For the sake of simplicity, we will assume $y_i$ corresponds to a single token in this illustration.
  • Figure 2: 2D plot comparing the accuracy (%) of LLaMA2 on Next Character Prediction (NCP-acc) with the Word Error Rate (WER %) of Whisper on a subset of languages from FLEURS. The plot serves as a point of reference for selecting languages that might benefit from SALSA. Specifically, we chose languages (colored in blue) that have high NCP-acc and medium to high WER using Whisper.