Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring the Redundancy of Decoder Layers in SpeechLLMs

Adel Moumen, Guangzhi Sun, Philip C Woodland

TL;DR

It is shown that the same blocks of layers are redundant across speech encoders, tasks and languages, indicating that a more global redundancy structure exists, enabling a single pruned and multi-tasks SpeechLLM backbone to be deployed.

Abstract

Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters. We study how much of this decoder capacity is actually needed for speech tasks. Across two LLM families and three scales (1-8B), we show that decoder redundancy is largely inherited from the pretrained LLM: text and speech inputs yield similar redundant blocks. We then measure excess capacity by pruning decoder layers and analysing post-pruning healing to increase robustness. Our findings show that 7-8B models retain good ASR performance with only 60% of decoder layers, and the same trend extends to smaller scales with reduced pruning tolerance. We then generalise to speech translation, and show that the same blocks of layers are redundant across speech encoders, tasks and languages, indicating that a more global redundancy structure exists, enabling a single pruned and multi-tasks SpeechLLM backbone to be deployed.

Measuring the Redundancy of Decoder Layers in SpeechLLMs

TL;DR

It is shown that the same blocks of layers are redundant across speech encoders, tasks and languages, indicating that a more global redundancy structure exists, enabling a single pruned and multi-tasks SpeechLLM backbone to be deployed.

Abstract

Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters. We study how much of this decoder capacity is actually needed for speech tasks. Across two LLM families and three scales (1-8B), we show that decoder redundancy is largely inherited from the pretrained LLM: text and speech inputs yield similar redundant blocks. We then measure excess capacity by pruning decoder layers and analysing post-pruning healing to increase robustness. Our findings show that 7-8B models retain good ASR performance with only 60% of decoder layers, and the same trend extends to smaller scales with reduced pruning tolerance. We then generalise to speech translation, and show that the same blocks of layers are redundant across speech encoders, tasks and languages, indicating that a more global redundancy structure exists, enabling a single pruned and multi-tasks SpeechLLM backbone to be deployed.
Paper Structure (15 sections, 2 equations, 2 figures, 2 tables)

This paper contains 15 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Speech Large Language Models
  3. SLAM Framework
  4. Measuring Decoder Redundancy
  5. Redundancy proxy
  6. Pruning algorithm
  7. Post-pruning healing
  8. Experimental Setup
  9. Datasets and evaluation
  10. Experimental Results
  11. Origin of decoder redundancy
  12. Pruning redundant layers on speech recognition
  13. Generalisation to speech translation
  14. Conclusions
  15. Generative AI Use Disclosure

Figures (2)

  • Figure 1: Angular distance between decoder layers $\ell$ and $\ell+n$, averaged over LibriSpeech dev-clean and dev-other. (a) Text-only input; (b) SLAM with frozen decoder; (c) SLAM with LoRA-adapted decoder; (d) SLAM Llama3.1-8B.
  • Figure 2: Relative WER degradation as a function of the fraction of decoder layers removed, for each evaluation set (columns) and model family (rows). The y-axis shows relative WER ($\Delta_{\mathrm{WER}}$) with respect to the unpruned baselines (Table \ref{['tab:asr_wer_main']}); a value of $2.0$ means twice the baseline WER. The grey dashed line at $0.25$ marks the maximum allowed relative degradation threshold used in our analysis.