OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models

TL;DR

OmniZip addresses the computational bottleneck of OmniLLMs caused by large audio–video token streams. It introduces a training-free audio-guided dynamic pruning approach that uses per-window audio retention scores $S_a(i)$, audio anchors, and an interleaved spatio-temporal compression (ISTC) module. The key contributions are the first omnimodal token compression method, a three-stage pipeline, and extensive benchmarks showing speedups up to 3x in prefilling and memory reductions around 10G with accuracy near baseline across 3B and 7B OmniLLMs. This work enables practical deployment of OmniLLMs for real-time audio–video understanding and provides a foundation for future efficiency techniques.

Abstract

Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding, wherein processing audio-video token sequences creates a significant computational bottleneck, however. Existing token compression methods have yet to accommodate this emerging need of jointly compressing multimodal tokens. To bridge this gap, we present OmniZip, a training-free, audio-guided audio-visual token-compression framework that optimizes multimodal token representation and accelerates inference. Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density, thereby dynamically guiding video token pruning and preserving cues from audio anchors enhanced by cross-modal similarity. For each time window, OmniZip compresses the video tokens using an interleaved spatio-temporal scheme. Extensive empirical results demonstrate the merits of OmniZip - it achieves 3.42X inference speedup and 1.4X memory reduction over other top-performing counterparts, while maintaining performance with no training.

Figures (8)

• Figure 1: (a): We introduce OmniZip, an audio-video token compression method tailored for efficient OmniLLMs. The key innovation is a "listen-to-prune" paradigm -- utilizing audio to dynamically guide video token pruning, complemented by a proposed compression module. (b): OmniZip achieves superior performance on various audio-video tasks on WorldSense hong2025worldsense, outperforming other methods. (c): Efficiency and performance comparison on WorldSense with Qwen2.5-Omni xu2025qwen2. OmniZip can achieve 2.51-3.42$\times$ wall-clock inference speedup (on an A6000 48G GPU), 1.4$\times$ memory reduction against other top-performing methods with almost the same performance.
• Figure 2: Audio tokens dominate attention heatmaps. Regular vertical bands aligned with audio-token positions indicate consistently higher attention to audio tokens, while many video tokens receive little attention, suggesting greater redundancy. Attention aggregates within time windows and decays across windows, indicating that audio and video tokens preferentially attend to short-range context within the same window. Moreover, deeper layers allocate less attention to raw audio and video tokens.
• Figure 3: Detailed overview of our OmniZip method. First, OmniZip computes an audio retention rate derived from dominant audio tokens to determine a dynamic pruning rate for the corresponding video tokens. Next, to preserve multimodal information, we uniformly sample audio anchors and merge with non-anchor tokens selected via cross-modal similarity. Finally, video tokens undergo interleaved spatio-temporal compression (ISTC), which alternately reduces temporal redundancy by merging cross-frame tokens and spatial redundancy by pruning intra-frame tokens. $\rho_a$ is the compression ratio of the audio token, $S_a(i)$ and $\rho_v(i)$ are the audio token retention ratio and video token compression ratio, in each time group, respectively.
• Figure 4: Ablation study on $\rho_a$ and $\rho_v$. All experiments illustrated in the figure were carried out on the Qwen2.5-Omni-7B model and the WorldSense benchmark. Left and Middle: We separately analyze the influence of varying $p_a$ and $p_v$ on model performance. In general, excessive pruning of either modality negatively impacts model performance. However, an appropriate balance of audio and video token pruning achieves the best effect. Right: Performance of our method vs. other methods in different compression ratios.
• Figure 5: Visualization of dynamic pruning ratios. The figure illustrates how audio token retention guides the allocation of video token pruning. Specifically, for time windows with low audio retention, we allocate a higher video pruning ratio, while maintaining a constant total pruning rate.