Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Segmentwise Pruning in Audio-Language Models

Marcel Gibier, Raphaël Duroselle, Pierre Serrano, Olivier Boeffard, Jean-François Bonastre

TL;DR

Audio-language models incur high compute costs from long audio inputs due to quadratic attention complexity. The paper introduces Segmentwise Top-K pruning, a temporally aware, inference-time token reduction method that segments the audio encoder output into $S$ intervals and retains within each segment $ig\lfloor K/S \big\rfloor$ tokens based on attention, reducing tokens by up to a factor of 4. Evaluations on Whisper-based ALMs across Clotho v2, AudioCaps, ClothoAQA, and MMAU show that retaining up to $25\%$ of tokens preserves performance and can even improve metrics like CIDEr in some cases (e.g., up to 23\% relative improvement on AudioCaps). The approach achieves notable efficiency gains, with substantial reductions in prefill time and consistent performance across tasks, suggesting practical benefits for long-audio processing and indicating directions for future work in adaptive token budgeting and temporal-aware pruning.

Abstract

Recent audio-language models have shown impressive performance across a wide range of audio tasks and are increasingly capable of handling long audio inputs. However, the computing costs in these models heavily depend on sequence length, which can become very large given the nature of audio data. In the vision-language domain, token pruning methods have proven effective in reducing token counts while preserving strong performance on standard benchmarks. In this work, we investigate the relevance and effectiveness of such token selection strategies in the context of audio-language models. We also improve them by proposing a lightweight strategy that takes the time dimension into account. While retaining only a quarter of the initial tokens, our approach results in a relative maximum decrease of 2% in CIDEr on Clotho v2 and a relative maximum decrease of 4% in accuracy on MMAU.

Segmentwise Pruning in Audio-Language Models

TL;DR

Audio-language models incur high compute costs from long audio inputs due to quadratic attention complexity. The paper introduces Segmentwise Top-K pruning, a temporally aware, inference-time token reduction method that segments the audio encoder output into intervals and retains within each segment tokens based on attention, reducing tokens by up to a factor of 4. Evaluations on Whisper-based ALMs across Clotho v2, AudioCaps, ClothoAQA, and MMAU show that retaining up to of tokens preserves performance and can even improve metrics like CIDEr in some cases (e.g., up to 23\% relative improvement on AudioCaps). The approach achieves notable efficiency gains, with substantial reductions in prefill time and consistent performance across tasks, suggesting practical benefits for long-audio processing and indicating directions for future work in adaptive token budgeting and temporal-aware pruning.

Abstract

Recent audio-language models have shown impressive performance across a wide range of audio tasks and are increasingly capable of handling long audio inputs. However, the computing costs in these models heavily depend on sequence length, which can become very large given the nature of audio data. In the vision-language domain, token pruning methods have proven effective in reducing token counts while preserving strong performance on standard benchmarks. In this work, we investigate the relevance and effectiveness of such token selection strategies in the context of audio-language models. We also improve them by proposing a lightweight strategy that takes the time dimension into account. While retaining only a quarter of the initial tokens, our approach results in a relative maximum decrease of 2% in CIDEr on Clotho v2 and a relative maximum decrease of 4% in accuracy on MMAU.

Paper Structure

This paper contains 16 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Attention Concentration on Few Tokens
  5. Comparison Methods
  6. Segmentwise Top-K Token Pruning
  7. Experiments and results
  8. Models
  9. Tasks and Benchmarks
  10. Implementation of experiments
  11. Results
  12. Further Analysis
  13. The importance of attention in the model’s response
  14. Efficiency Analysis
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Audio token embeddings are color-coded by aggregate incoming attention. Segmentwise Top-K, with $S=\frac{N}{3}$ and $\lfloor K/S \rfloor=1$ ($S$ = number of segments, $K$ = total tokens retained), selects one token every three tokens from an audio sequence of length $N$.
  • Figure 2: Max-normalized attention received by each token for a 10-second Clotho v2 audio sample. In orange: the 10 most important tokens. Rectangles indicate groups of important tokens that are very close to each other.