Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models

Anmin Wang, Nan Zhang, Wei Tao, Xiaoyang Qu, Guokuan Li, Jiguang Wan, Jianzong Wang

TL;DR

Triage addresses the heavy computational burden of video reasoning in Vision-Language Models by reframing inference as a hierarchical budgeting problem. It first performs Frame-Level Budgeting to select a compact set of keyframes using a frame importance score that combines visual dynamics and query relevance, producing a prior for downstream processing. It then executes Token-Level Budgeting, allocating a constrained token budget into high-relevance Core Tokens and diverse Context Tokens via a batched MMR method guided by cross-attention, with budget distribution across frames informed by frame importance. Experiments across multiple backbones and benchmarks show that Triage yields significant speedups and memory savings while maintaining or surpassing state-of-the-art baselines, demonstrating a practical, plug-and-play solution for efficient video reasoning in VLMs.

Abstract

Vision-Language Models (VLMs) face significant computational challenges in video processing due to massive data redundancy, which creates prohibitively long token sequences. To address this, we introduce Triage, a training-free, plug-and-play framework that reframes video reasoning as a resource allocation problem via hierarchical visual budgeting. Its first stage, Frame-Level Budgeting, identifies keyframes by evaluating their visual dynamics and relevance, generating a strategic prior based on their importance scores. Guided by this prior, the second stage, Token-Level Budgeting, allocates tokens in two phases: it first secures high-relevance Core Tokens, followed by diverse Context Tokens selected with an efficient batched Maximal Marginal Relevance (MMR) algorithm. Extensive experiments demonstrate that Triage improves inference speed and reduces memory footprint, while maintaining or surpassing the performance of baselines and other methods on various video reasoning benchmarks.

Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models

TL;DR

Triage addresses the heavy computational burden of video reasoning in Vision-Language Models by reframing inference as a hierarchical budgeting problem. It first performs Frame-Level Budgeting to select a compact set of keyframes using a frame importance score that combines visual dynamics and query relevance, producing a prior for downstream processing. It then executes Token-Level Budgeting, allocating a constrained token budget into high-relevance Core Tokens and diverse Context Tokens via a batched MMR method guided by cross-attention, with budget distribution across frames informed by frame importance. Experiments across multiple backbones and benchmarks show that Triage yields significant speedups and memory savings while maintaining or surpassing state-of-the-art baselines, demonstrating a practical, plug-and-play solution for efficient video reasoning in VLMs.

Abstract

Vision-Language Models (VLMs) face significant computational challenges in video processing due to massive data redundancy, which creates prohibitively long token sequences. To address this, we introduce Triage, a training-free, plug-and-play framework that reframes video reasoning as a resource allocation problem via hierarchical visual budgeting. Its first stage, Frame-Level Budgeting, identifies keyframes by evaluating their visual dynamics and relevance, generating a strategic prior based on their importance scores. Guided by this prior, the second stage, Token-Level Budgeting, allocates tokens in two phases: it first secures high-relevance Core Tokens, followed by diverse Context Tokens selected with an efficient batched Maximal Marginal Relevance (MMR) algorithm. Extensive experiments demonstrate that Triage improves inference speed and reduces memory footprint, while maintaining or surpassing the performance of baselines and other methods on various video reasoning benchmarks.
Paper Structure (10 sections, 3 equations, 2 figures, 2 tables)

This paper contains 10 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Frame-Level Budgeting
  4. Token-Level Budgeting
  5. Experiments
  6. Experimental Setup
  7. Main Results
  8. Ablation Study and Analysis
  9. Conclusion
  10. Acknowledgments

Figures (2)

  • Figure 1: An overview of the Triage framework.
  • Figure 2: Performance-efficiency trade-off and hyperparameter sensitivity analysis for Triage.