Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval

Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai "Helen" Li, Yiran Chen

TL;DR

This work tackles long-context speech information retrieval (SIR) for Speech LLMs by introducing SPIRAL, a ~90-second benchmark, and a training-free token pruning method called SpeechPrune. SpeechPrune uses a two-phase approach based on speech-text similarity and approximate attention to drastically reduce input tokens while preserving essential content, enabling efficient long-form processing. On Qwen-2 Audio and DiVA, SpeechPrune yields substantial accuracy gains at modest pruning rates (up to 29% over the original and 47% over random pruning) and maintains performance even at high pruning levels, with significant reductions in TFLOPs and memory. The work demonstrates that token-level pruning is a viable path for scalable long-form speech understanding across diverse models and benchmarks.

Abstract

We introduce Speech Information Retrieval (SIR), a new long-context task for Speech Large Language Models (Speech LLMs), and present SPIRAL, a 1,012-sample benchmark testing models' ability to extract critical details from approximately 90-second spoken inputs. While current Speech LLMs excel at short-form tasks, they struggle with the computational and representational demands of longer audio sequences. To address this limitation, we propose SpeechPrune, a training-free token pruning strategy that uses speech-text similarity and approximated attention scores to efficiently discard irrelevant tokens. In SPIRAL, SpeechPrune achieves accuracy improvements of 29% and up to 47% over the original model and the random pruning model at a pruning rate of 20%, respectively. SpeechPrune can maintain network performance even at a pruning level of 80%. This approach highlights the potential of token-level pruning for efficient and scalable long-form speech understanding.

SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval

TL;DR

This work tackles long-context speech information retrieval (SIR) for Speech LLMs by introducing SPIRAL, a ~90-second benchmark, and a training-free token pruning method called SpeechPrune. SpeechPrune uses a two-phase approach based on speech-text similarity and approximate attention to drastically reduce input tokens while preserving essential content, enabling efficient long-form processing. On Qwen-2 Audio and DiVA, SpeechPrune yields substantial accuracy gains at modest pruning rates (up to 29% over the original and 47% over random pruning) and maintains performance even at high pruning levels, with significant reductions in TFLOPs and memory. The work demonstrates that token-level pruning is a viable path for scalable long-form speech understanding across diverse models and benchmarks.

Abstract

We introduce Speech Information Retrieval (SIR), a new long-context task for Speech Large Language Models (Speech LLMs), and present SPIRAL, a 1,012-sample benchmark testing models' ability to extract critical details from approximately 90-second spoken inputs. While current Speech LLMs excel at short-form tasks, they struggle with the computational and representational demands of longer audio sequences. To address this limitation, we propose SpeechPrune, a training-free token pruning strategy that uses speech-text similarity and approximated attention scores to efficiently discard irrelevant tokens. In SPIRAL, SpeechPrune achieves accuracy improvements of 29% and up to 47% over the original model and the random pruning model at a pruning rate of 20%, respectively. SpeechPrune can maintain network performance even at a pruning level of 80%. This approach highlights the potential of token-level pruning for efficient and scalable long-form speech understanding.

Paper Structure

This paper contains 14 sections, 13 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Speech Information Retrieval
  3. Task Formulation
  4. Benchmark Construction
  5. Quality Assessment
  6. SpeechPrune
  7. Preliminaries
  8. SpeechPrune Methodology
  9. Experiments
  10. Main Experiments
  11. Qualitative Analysis
  12. Ablation Studies
  13. Generalization Analysis
  14. Conclusion

Figures (3)

  • Figure 1: The proposed SpeechPrune, with two phases of token pruning.
  • Figure 2: Qualitative analysis of token embeddings via t-SNE visualization, where high-dimensional embeddings are projected into 2D space for visualization. (a) SpeechPrune (b) Random pruning. Gray, blue, and red points represent pruned audio tokens, preserved audio tokens, and text tokens, respectively.
  • Figure 3: Ablation study comparing different pruning strategies on SPIRAL-H dataset. The plot shows the accuracy of three approaches: first phase only, second phase only, and SpeechPrune. The dotted line shows the accuracy when using the complete, unpruned set of input tokens.