IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs

TL;DR

IQViC introduces a transformer-based visual compressor that performs in-context, question-adaptive compression to create a compact, memory-efficient context for long-term video understanding. The framework comprises a visual encoder, a question-conditioned visual compressor, a sequential context memory with a lightweight temporal compressor, and an LLM-based decoder, trained in two steps with LoRA on image QA and then video QA data. Empirical results on InfiniBench-Vision show IQViC surpassing state-of-the-art long-term VQA methods while using far fewer memory tokens, and it also matches or exceeds short-term VQA performance with a simpler memory architecture. The authors also present InfiniBench-Vision, a frame-only subset of InfiniBench designed to evaluate long-term video understanding from frames alone, and discuss limitations and future directions toward end-to-end training and multimodal extensions.

Abstract

With the increasing complexity of video data and the need for more efficient long-term temporal understanding, existing long-term video understanding methods often fail to accurately capture and analyze extended video sequences. These methods typically struggle to maintain performance over longer durations and to handle the intricate dependencies within the video content. To address these limitations, we propose a simple yet effective large multi-modal model framework for long-term video understanding that incorporates a novel visual compressor, the In-context, Question Adaptive Visual Compressor (IQViC). The key idea, inspired by humans' selective attention and in-context memory mechanisms, is to introduce a novel visual compressor and incorporate efficient memory management techniques to enhance long-term video question answering. Our framework utilizes IQViC, a transformer-based visual compressor, enabling question-conditioned in-context compression, unlike existing methods that rely on full video visual features. This selectively extracts relevant information, significantly reducing memory token requirements. Through extensive experiments on a new dataset based on InfiniBench for long-term video understanding, and standard benchmarks used for existing methods' evaluation, we demonstrate the effectiveness of our proposed IQViC framework and its superiority over state-of-the-art methods in terms of video understanding accuracy and memory efficiency.

Figures (10)

• Figure 1: Conventional methods
• Figure 2: Proposed method
• Figure 4: The architecture of the proposed IQViC framework in detail. The visual encoder extracts visual features from each video frame and projects them into visual embeddings. Then, the proposed visual compressor, IQViC, transforms the visual embeddings into context embeddings conditioned on the text prompt. The context memory stores the compressed context embeddings sequentially with a predefined capacity. Finally, the LLM generates answer to the text prompt. The snowflake symbol represents components with fixed parameters, while the flame icon indicates the modules that are trained.
• Figure 5: Step 1: Training the visual compressor and embedding module
• Figure 6: Step 2: Finetuning the LLM