Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two-Stage Acoustic Adaptation with Gated Cross-Attention Adapters for LLM-Based Multi-Talker Speech Recognition

Hao Shi, Yuan Gao, Xugang Lu, Tatsuya Kawahara

Abstract

Large Language Models (LLMs) are strong decoders for Serialized Output Training (SOT) in two-talker Automatic Speech Recognition (ASR), yet their performance degrades substantially in challenging conditions such as three-talker mixtures. A key limitation is that current systems inject acoustic evidence only through a projected prefix, which can be lossy and imperfectly aligned with the LLM input space, providing insufficient fine-grained grounding during decoding. Addressing this limitation is crucial for robust multi-talker ASR, especially in three-talker mixtures. This paper improves LLM-based multi-talker ASR by explicitly injecting talker-aware acoustic evidence into the decoder. We first revisit Connectionist Temporal Classification (CTC)-derived prefix prompting and compare three variants with increasing acoustic content. The CTC information is obtained using the serialized CTC proposed in our previous works. While acoustic-enriched prompts outperform the SOT-only baseline, prefix-only conditioning remains inadequate for three-talker mixtures. We therefore propose a lightweight gated residual cross-attention adapter and design a two-stage acoustic adaptation framework based on low-rank updates (LoRA). In Stage 1, we insert gated cross-attention adapters after the self-attention sub-layer to stably inject acoustic embeddings as external memory. In Stage 2, we refine both the cross-attention adapters and the pretrained LLM's self-attention projections using parameter-efficient LoRA, improving robustness for large backbones under limited data; the learned updates are merged into the base weights for inference. Experiments on Libri2Mix/Libri3Mix under clean and noisy conditions show consistent gains, with particularly large improvements in three-talker settings.

Two-Stage Acoustic Adaptation with Gated Cross-Attention Adapters for LLM-Based Multi-Talker Speech Recognition

Abstract

Large Language Models (LLMs) are strong decoders for Serialized Output Training (SOT) in two-talker Automatic Speech Recognition (ASR), yet their performance degrades substantially in challenging conditions such as three-talker mixtures. A key limitation is that current systems inject acoustic evidence only through a projected prefix, which can be lossy and imperfectly aligned with the LLM input space, providing insufficient fine-grained grounding during decoding. Addressing this limitation is crucial for robust multi-talker ASR, especially in three-talker mixtures. This paper improves LLM-based multi-talker ASR by explicitly injecting talker-aware acoustic evidence into the decoder. We first revisit Connectionist Temporal Classification (CTC)-derived prefix prompting and compare three variants with increasing acoustic content. The CTC information is obtained using the serialized CTC proposed in our previous works. While acoustic-enriched prompts outperform the SOT-only baseline, prefix-only conditioning remains inadequate for three-talker mixtures. We therefore propose a lightweight gated residual cross-attention adapter and design a two-stage acoustic adaptation framework based on low-rank updates (LoRA). In Stage 1, we insert gated cross-attention adapters after the self-attention sub-layer to stably inject acoustic embeddings as external memory. In Stage 2, we refine both the cross-attention adapters and the pretrained LLM's self-attention projections using parameter-efficient LoRA, improving robustness for large backbones under limited data; the learned updates are merged into the base weights for inference. Experiments on Libri2Mix/Libri3Mix under clean and noisy conditions show consistent gains, with particularly large improvements in three-talker settings.

Paper Structure

This paper contains 35 sections, 33 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. LLMs as ASR Decoders
  4. Speech encoding and downsampling
  5. Projection and LLM decoding
  6. Training objective
  7. SOT for Multi-Talker ASR
  8. Serialized CTC for Multi-Talker ASR
  9. Serialized CTC loss (SerCTC)
  10. Overall training objective
  11. SOP for LLM-Based Multi-Talker ASR
  12. Architecture
  13. SOP construction via separator and serialized CTC
  14. Revisiting Prefix Prompting with CTC Cues for Multi-Talker ASR
  15. Encoder-Side Serialized Talker Information Extraction
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Final model architecture. Projected acoustic embeddings are provided as external memory and injected into the LLM decoder via the proposed cross-attention adapter.
  • Figure 2: Overview and motivation of the proposed two-stage adaptation. Stage 0 adapts the LLM decoder with self-attention LoRA (baseline). Stage 1 introduces a gated residual cross-attention adapter to inject talker-aware acoustic evidence. Stage 2 refines the cross-attention adapters and self-attention with LoRA to improve robustness and reduce hyperparameter sensitivity under large backbones and limited data.