HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding

TL;DR

HIPPO tackles the latency bottleneck of video-LLM inference by introducing semantic-aware token preservation and video parallel speculative decoding. The approach preserves semantically informative visual tokens at high pruning ratios and overlaps draft generation with target verification to hide draft overhead, achieving up to 3.51x speedup with higher mean accepted tokens. Extensive experiments across four video-LLMs and six benchmarks demonstrate robustness, while ablations validate the contribution of global-local scoring and parallelism. This framework advances practical, lossless acceleration for video-LLMs by integrating cross-modal semantics with parallel decoding strategies.

Abstract

Speculative decoding (SD) has emerged as a promising approach to accelerate LLM inference without sacrificing output quality. Existing SD methods tailored for video-LLMs primarily focus on pruning redundant visual tokens to mitigate the computational burden of massive visual inputs. However, existing methods do not achieve inference acceleration comparable to text-only LLMs. We observe from extensive experiments that this phenomenon mainly stems from two limitations: (i) their pruning strategies inadequately preserve visual semantic tokens, degrading draft quality and acceptance rates; (ii) even with aggressive pruning (e.g., 90% visual tokens removed), the draft model's remaining inference cost limits overall speedup. To address these limitations, we propose HIPPO, a general holistic-aware parallel speculative decoding framework. Specifically, HIPPO proposes (i) a semantic-aware token preservation method, which fuses global attention scores with local visual semantics to retain semantic information at high pruning ratios; (ii) a video parallel SD algorithm that decouples and overlaps draft generation and target verification phases. Experiments on four video-LLMs across six benchmarks demonstrate HIPPO's effectiveness, yielding up to 3.51x speedup compared to vanilla auto-regressive decoding.

Figures (20)

• Figure 1: Visualization of video frames under 90% pruning ratio. (i) The upper part attention-only pruning suffers from the position bias phenomenon, where inflated attention weights are allocated to non-informative boundary tokens (highlighted by orange boxes). (ii) The lower part semantic-aware token preservation preserves informative visual tokens (highlighted by green boxes), retaining the core action despite the high pruning ratio.
• Figure 2: Statistics of per-frame positional bias.
• Figure 3: Performance comparison of three visual token inputs: random 10% sampling, target-model-guided top-10% attention selection, and full visual tokens.
• Figure 4: Statistics of average running time ratio of the draft and target model.
• Figure 5: An overview of HIPPO. (A) Semantic-aware Token Preservation: A holistic scoring mechanism integrates global semantic relevance, inter-frame temporal redundancy, and intra-frame spatial redundancy to select a compact subset of informative visual tokens, mitigating position bias. (B) Video Parallel Speculative Decoding: Synchronous Draft-Target Prefill generates initial draft tokens while the target model processes input, followed by an Adaptive Decoding Strategy that dynamically switches between optimistic post-verification and conservative pre-verification modes based on draft quality to maximize parallelism.