Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference

Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, Joon Son Chung

TL;DR

FastAV tackles the high computational cost of AV-LLMs by introducing attention rollout-guided two-stage token pruning. The method globally prunes tokens in mid layers using rollout to remove less informative tokens, then fine-prunes with last-query token scores using $s^l = \text{mean}_{h}(\text{softmax}(Q^l_{\text{last}} (K^l)^T))$ to minimize harm to next-token prediction, all without full attention maps and with compatibility to FlashAttention. On VideoLLaMA2 and video-SALMONN2, FastAV achieves over 40% FLOPs reduction while maintaining or improving accuracy across AVQA, MUSIC-AVQA, and AVHBench, including dramatic reductions in audio tokens (e.g., from 1496 to 10). This enables efficient processing of long multimodal inputs and demonstrates practical impact for deploying AV-LLMs in real-time or resource-constrained settings.

Abstract

In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance.

FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference

TL;DR

FastAV tackles the high computational cost of AV-LLMs by introducing attention rollout-guided two-stage token pruning. The method globally prunes tokens in mid layers using rollout to remove less informative tokens, then fine-prunes with last-query token scores using to minimize harm to next-token prediction, all without full attention maps and with compatibility to FlashAttention. On VideoLLaMA2 and video-SALMONN2, FastAV achieves over 40% FLOPs reduction while maintaining or improving accuracy across AVQA, MUSIC-AVQA, and AVHBench, including dramatic reductions in audio tokens (e.g., from 1496 to 10). This enables efficient processing of long multimodal inputs and demonstrates practical impact for deploying AV-LLMs in real-time or resource-constrained settings.

Abstract

In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance.
Paper Structure (8 sections, 4 equations, 4 figures, 4 tables)

This paper contains 8 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. AV-LLM Inference
  4. FastAV
  5. Experiments
  6. Experimental setup
  7. Experimental Results
  8. Conclusion

Figures (4)

  • Figure 1: Attention rollout at the 14th layer in VideoLLaMA2 cheng2024videollama and video-SALMONN2 tang2025video. Accumulated attention concentrates on earlier tokens, highlighting their pivotal role in carrying the most essential information.
  • Figure 2: Attention rollout and weights across layers in VideoLLaMA2 cheng2024videollama. Attention rollout reveals a progressive focus on earlier tokens, stabilizing around the middle layers and persisting in deeper layers, whereas raw attention weights alone do not exhibit such a clear pattern.
  • Figure 3: Overview of the FastAV framework. FastAV starts with the full input context and reduces computation through two-stage pruning. In the middle layer, global pruning removes later tokens guided by attention rollout. In subsequent layers, fine-grained pruning discards the least important P% of remaining tokens based on last-query token analysis.
  • Figure 4: Layerwise accuracy of VideoLLaMA2 on AVHBench subtasks. The middle layer are chosen to balance performance between AV hallucination and AV matching tasks.