Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs and Speech: Integration vs. Combination

Robin Schmitt, Albert Zeyer, Mohammad Zeineldeen, Ralf Schlüter, Hermann Ney

Abstract

In this work, we study how to best utilize pre-trained LLMs for automatic speech recognition. Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion. For tight integration, we provide ablations on the effect of different label units, fine-tuning strategies, LLM sizes and pre-training data, attention interfaces, encoder downsampling, text prompts, and length normalization. Additionally, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations for this joint recognition. For shallow fusion, we investigate the effect of fine-tuning the LLM on the transcriptions using different label units, and we compare rescoring AM hypotheses to single-pass recognition with label-wise or delayed fusion of AM and LLM scores. We train on Librispeech and Loquacious and evaluate our models on the HuggingFace ASR leaderboard.

LLMs and Speech: Integration vs. Combination

Abstract

In this work, we study how to best utilize pre-trained LLMs for automatic speech recognition. Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion. For tight integration, we provide ablations on the effect of different label units, fine-tuning strategies, LLM sizes and pre-training data, attention interfaces, encoder downsampling, text prompts, and length normalization. Additionally, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations for this joint recognition. For shallow fusion, we investigate the effect of fine-tuning the LLM on the transcriptions using different label units, and we compare rescoring AM hypotheses to single-pass recognition with label-wise or delayed fusion of AM and LLM scores. We train on Librispeech and Loquacious and evaluate our models on the HuggingFace ASR leaderboard.
Paper Structure (34 sections, 13 equations, 1 figure, 25 tables)

This paper contains 34 sections, 13 equations, 1 figure, 25 tables.

Table of Contents

  1. Introduction & Related Work
  2. Shallow Fusion Baseline
  3. Acoustic Encoder.
  4. Acoustic CTC output.
  5. Language model.
  6. Training
  7. Recognition
  8. Integrated Models
  9. Encoder and CTC output.
  10. Adapter.
  11. Decoder and attention interface variants.
  12. Training
  13. Recognition
  14. Experiments
  15. From Scratch vs. Fine-tuning
  16. ...and 19 more sections

Figures (1)

  • Figure 1: Self-attention weights of prefix LLM. Model is initialized with the baseline AED encoder and the Qwen2 0.5B decoder and fine-tuned for 1 epoch on Loquacious. From bottom to top: layer 24 head 7, layer 5 head 8, layer 13 head 13. The X-axis shows the key/value positions for corresponding inputs, and the Y-axis shows the query positions for the decoder output (omitting part of the sequence for better visibility).