Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors

Rao Ma, Tongzhou Chen, Kartik Audhkhasi, Bhuvana Ramabhadran

TL;DR

LegoSLM introduces a modular framework that connects pre-trained speech encoders with large language models by training the encoder with CTC loss over the LLM vocabulary and reconstructing speech embeddings from the resulting posteriors. The reconstructed embeddings are fused with LLM inputs, enabling a decoder-only, zero-shot capable integration that preserves encoder–LLM modularity and reduces information loss relative to cascaded approaches. Temperature-based control of the CTC softmax, plus top-K variants, provides a tunable balance between acoustic and linguistic priors across ASR and speech translation tasks. Empirical results on MLS, LibriSpeech, and CoVoST 2 demonstrate competitive ASR performance, strong zero-shot modularity, and superior translation quality, highlighting LegoSLM’s potential for flexible, scalable spoken-language understanding and translation. The work points to broader applicability in tasks such as speech summarization and domain-adaptive spoken language processing.

Abstract

Recently, large-scale pre-trained speech encoders and Large Language Models (LLMs) have been released, which show state-of-the-art performance on a range of spoken language processing tasks including Automatic Speech Recognition (ASR). To effectively combine both models for better performance, continuous speech prompts, and ASR error correction have been adopted. However, these methods are prone to suboptimal performance or are inflexible. In this paper, we propose a new paradigm, LegoSLM, that bridges speech encoders and LLMs using the ASR posterior matrices. The speech encoder is trained to generate Connectionist Temporal Classification (CTC) posteriors over the LLM vocabulary, which are used to reconstruct pseudo-audio embeddings by computing a weighted sum of the LLM input embeddings. These embeddings are concatenated with text embeddings in the LLM input space. Using the well-performing USM and Gemma models as an example, we demonstrate that our proposed LegoSLM method yields good performance on both ASR and speech translation tasks. By connecting USM with Gemma models, we can get an average of 49% WERR over the USM-CTC baseline on 8 MLS testsets. The trained model also exhibits modularity in a range of settings -- after fine-tuning the Gemma model weights, the speech encoder can be switched and combined with the LLM in a zero-shot fashion. Additionally, we propose to control the decode-time influence of the USM and LLM using a softmax temperature, which shows effectiveness in domain adaptation.

LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors

TL;DR

LegoSLM introduces a modular framework that connects pre-trained speech encoders with large language models by training the encoder with CTC loss over the LLM vocabulary and reconstructing speech embeddings from the resulting posteriors. The reconstructed embeddings are fused with LLM inputs, enabling a decoder-only, zero-shot capable integration that preserves encoder–LLM modularity and reduces information loss relative to cascaded approaches. Temperature-based control of the CTC softmax, plus top-K variants, provides a tunable balance between acoustic and linguistic priors across ASR and speech translation tasks. Empirical results on MLS, LibriSpeech, and CoVoST 2 demonstrate competitive ASR performance, strong zero-shot modularity, and superior translation quality, highlighting LegoSLM’s potential for flexible, scalable spoken-language understanding and translation. The work points to broader applicability in tasks such as speech summarization and domain-adaptive spoken language processing.

Abstract

Recently, large-scale pre-trained speech encoders and Large Language Models (LLMs) have been released, which show state-of-the-art performance on a range of spoken language processing tasks including Automatic Speech Recognition (ASR). To effectively combine both models for better performance, continuous speech prompts, and ASR error correction have been adopted. However, these methods are prone to suboptimal performance or are inflexible. In this paper, we propose a new paradigm, LegoSLM, that bridges speech encoders and LLMs using the ASR posterior matrices. The speech encoder is trained to generate Connectionist Temporal Classification (CTC) posteriors over the LLM vocabulary, which are used to reconstruct pseudo-audio embeddings by computing a weighted sum of the LLM input embeddings. These embeddings are concatenated with text embeddings in the LLM input space. Using the well-performing USM and Gemma models as an example, we demonstrate that our proposed LegoSLM method yields good performance on both ASR and speech translation tasks. By connecting USM with Gemma models, we can get an average of 49% WERR over the USM-CTC baseline on 8 MLS testsets. The trained model also exhibits modularity in a range of settings -- after fine-tuning the Gemma model weights, the speech encoder can be switched and combined with the LLM in a zero-shot fashion. Additionally, we propose to control the decode-time influence of the USM and LLM using a softmax temperature, which shows effectiveness in domain adaptation.
Paper Structure (32 sections, 7 equations, 5 figures, 16 tables)

This paper contains 32 sections, 7 equations, 5 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Modular ASR Approach
  4. ASR Error Correction
  5. LLM with Speech Prompts
  6. Methodology
  7. Structure of Speech Encoder
  8. The LegoSLM Connection Method
  9. Discussion of the CTC Blank Token
  10. Zero-shot System Combination
  11. AM/LM Spectrum Control
  12. Model Variants
  13. Experimental Setup
  14. Models
  15. Baselines
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Comparison of different connection methods: ASR error correction (in green), speech prompts (in orange), and the proposed LegoSLM (in red).
  • Figure 2: Depiction of the proposed LegoSLM method. The speech embeddings are reconstructed using ASR CTC posteriors and the LLM embedding table.
  • Figure 3: Illustration of the zero-shot system combination test where speech encoders and LLMs trained in different setups are seamlessly combined.
  • Figure 4: Effect of changing the temperature value in LegoSLM. The speech encoder and LLM are trained on different setups. Top: USM-CTC (lbs) + Gemma (mls-en). Bottom: USM-CTC (mls-en) + Gemma (lbs).
  • Figure 5: Effect of changing the temperature value in LegoSLM on the CoVoST 2 en$\rightarrow$de speech translation task. For each experimental configuration, the speech encoder and LLM are trained under different setups and combined in a zero-shot fashion. Left: USM-CTC (public) + Gemma (multi). Right: USM-CTC (lbs) + Gemma (multi).