Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs

Vaggelis Dorovatas, Soroush Seifi, Gunshi Gupta, Rahaf Aljundi

TL;DR

This paper tackles streaming long-video understanding for Video-LLMs by proposing a training-free pipeline, rLiVS, that compresses each short clip via LLM-attention-guided token selection (retaining only ≈$6\%$ of visual tokens per clip), reuses past selected tokens as a recurrent memory, and answers through caption-based retrieval and reasoning. The method is designed to be agnostic to the underlying video-LLM and requires no additional training, achieving state-of-the-art results on streaming benchmarks while reducing memory usage by substantial margins (e.g., ~11GB less VRAM for the same backbone). By leveraging caption embeddings for long-context reasoning and employing MMRetrieval with MMR to balance relevance and diversity, rLiVS demonstrates robust performance across offline and online datasets and with different backbones (e.g., 7B and 0.5B). The work highlights the practicality of efficient, memory-conscious long-video understanding and sets a strong, training-free baseline for future streaming Video-LLMs, with potential extensions to summarization, retrieval, and tracking tasks.

Abstract

Video Large Language Models (Video-LLMs) excel at understanding videos in-context, provided they have full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95% of unimportant visual tokens with minimal performance loss; 2) Recurrent processing of past selected tokens to generate temporally coherent understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.

Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs

TL;DR

This paper tackles streaming long-video understanding for Video-LLMs by proposing a training-free pipeline, rLiVS, that compresses each short clip via LLM-attention-guided token selection (retaining only ≈ of visual tokens per clip), reuses past selected tokens as a recurrent memory, and answers through caption-based retrieval and reasoning. The method is designed to be agnostic to the underlying video-LLM and requires no additional training, achieving state-of-the-art results on streaming benchmarks while reducing memory usage by substantial margins (e.g., ~11GB less VRAM for the same backbone). By leveraging caption embeddings for long-context reasoning and employing MMRetrieval with MMR to balance relevance and diversity, rLiVS demonstrates robust performance across offline and online datasets and with different backbones (e.g., 7B and 0.5B). The work highlights the practicality of efficient, memory-conscious long-video understanding and sets a strong, training-free baseline for future streaming Video-LLMs, with potential extensions to summarization, retrieval, and tracking tasks.

Abstract

Video Large Language Models (Video-LLMs) excel at understanding videos in-context, provided they have full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95% of unimportant visual tokens with minimal performance loss; 2) Recurrent processing of past selected tokens to generate temporally coherent understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.
Paper Structure (40 sections, 6 equations, 3 figures, 13 tables, 1 algorithm)

This paper contains 40 sections, 6 equations, 3 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Attention based Visual Token Selection
  5. Short Clip Processing.
  6. Long Video Recurrent Processing.
  7. Efficient Video Question Answering
  8. Experiments
  9. Evaluation Benchmarks.
  10. Baselines.
  11. Evaluation Metrics.
  12. Implementation Details.
  13. Results
  14. Evaluation of Visual Token Selection Strategies on Short Video Datasets
  15. Offline Video Question Answering Datasets
  16. ...and 25 more sections

Figures (3)

  • Figure 1: Pipeline of rLiVS: (Left): Long videos are broken down into short visual clips which are processed in a streaming fashion. A short clip is pre-pended with selected visual tokens from past clips in the stream to generate a textual description. History is accumulated by selecting a subset of tokens from the short clip based on attention scores. (Right): Given a query on the past video, the accumulated textual descriptions over a long video are compared to the query embedding and the most similar subset is input to the LLM (within the videoLLM) for question-answering.
  • Figure 2: Distribution of cosine similarities between visual tokens and queries (left), and between captions and queries (right), averaged over videos and questions in RVS-Movie.
  • Figure : Streaming Video Processing and Query Answering with rLiVS