Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing Discrete and Continuous Space LLMs for Speech Recognition

Yaoxun Xu, Shi-Xiong Zhang, Jianwei Yu, Zhiyong Wu, Dong Yu

TL;DR

This work systematically compares discrete and continuous speech representations within LLM-based automatic speech recognition, organizing representations into four categories based on supervision and data form, and evaluating them with two language-model backbones: a Joint-Training-From-Scratch Language Model (JTFS LM) and pretrained LLaMA2-7b. By employing specialized encoders (HuBERT and Whisper) and a range of modeling and training strategies, the study demonstrates that continuous representations generally yield lower WER than discrete ones under JTFS LM, and that LLaMA2 with TextInput strategies can achieve competitive, state-of-the-art open-source results on LibriSpeech (best around 1.6% clean/3.0% other). The key contributions include the first thorough cross-comparison of discrete vs continuous speech representations in LLM-based ASR, a detailed modeling blueprint for each combination, and open-source results that advance ASR and NLP research. The findings underscore the importance of representation fidelity, encoder choice, and pretraining benefits for effective integration of speech with large language models, with practical implications for building cost-efficient yet accurate speech interfaces.

Abstract

This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69\% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.

Comparing Discrete and Continuous Space LLMs for Speech Recognition

TL;DR

This work systematically compares discrete and continuous speech representations within LLM-based automatic speech recognition, organizing representations into four categories based on supervision and data form, and evaluating them with two language-model backbones: a Joint-Training-From-Scratch Language Model (JTFS LM) and pretrained LLaMA2-7b. By employing specialized encoders (HuBERT and Whisper) and a range of modeling and training strategies, the study demonstrates that continuous representations generally yield lower WER than discrete ones under JTFS LM, and that LLaMA2 with TextInput strategies can achieve competitive, state-of-the-art open-source results on LibriSpeech (best around 1.6% clean/3.0% other). The key contributions include the first thorough cross-comparison of discrete vs continuous speech representations in LLM-based ASR, a detailed modeling blueprint for each combination, and open-source results that advance ASR and NLP research. The findings underscore the importance of representation fidelity, encoder choice, and pretraining benefits for effective integration of speech with large language models, with practical implications for building cost-efficient yet accurate speech interfaces.

Abstract

This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69\% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.
Paper Structure (11 sections, 2 equations, 2 figures, 2 tables)

This paper contains 11 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. LLM based Speech Recogntion
  3. Discrete V.S. Continuous Speech Representations
  4. Discrete V.S. Continuous Modeling
  5. Training Loss
  6. Experiment
  7. Result
  8. Results on joint-training-from-scratch language model
  9. Results on LLaMA2
  10. Conclusion
  11. Acknowledgement

Figures (2)

  • Figure 1: Overall architectures for continuous and discrete speech encoders. Figures (b) to (e) illustrate four distinct speech encoders, each extracting a different type of speech representation.
  • Figure 2: Model design for discrete and continuous scenarios. In Figure 2(b), dashed lines show the data flow for the JTFS LM, and solid lines for the LLaMA2 model