PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

TL;DR

This work tackles the challenge of unified short- and long-video understanding by identifying redundancy as a core bottleneck and proposing PPLLaVA, a prompt-guided pooling framework. It combines Fine-grained Vision-Prompt Alignment, a 3D convolution-style Prompt-Guided Pooling, and CLIP Context Extension to compress visual tokens by $>90\%$ while preserving instruction-relevant information, enabling flexible output sizes and high throughput with a $1024$-token visual context. Built on image-domain LLaVA foundations, it integrates interleaved training and Direct Preference Optimization (DPO) to reduce hallucinations and transfer from image to video domains, achieving state-of-the-art results across diverse video benchmarks from seconds to hours. The approach demonstrates strong performance on long videos (Video-MME) and competitive results on image benchmarks, highlighting significant efficiency gains and practical applicability for multimodal dialogue and reasoning.

Abstract

The past year has witnessed the significant advancement of video-based large language models. However, the challenge of developing a unified model for both short and long video understanding remains unresolved. Most existing video LLMs cannot handle hour-long videos, while methods custom for long videos tend to be ineffective for shorter videos and images. In this paper, we identify the key issue as the redundant content in videos. To address this, we propose a novel pooling strategy that simultaneously achieves token compression and instruction-aware visual feature aggregation. Our model is termed Prompt-guided Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three core components: the CLIP-based visual-prompt alignment that extracts visual information relevant to the user's instructions, the prompt-guided pooling that compresses the visual sequence to arbitrary scales using convolution-style pooling, and the clip context extension designed for lengthy prompt common in visual dialogue. Moreover, our codebase also integrates the most advanced video Direct Preference Optimization (DPO) and visual interleave training. Extensive experiments have validated the performance of our model. With superior throughput and only 1024 visual context, PPLLaVA achieves better results on image benchmarks as a video LLM, while achieving state-of-the-art performance across various video benchmarks, excelling in tasks ranging from caption generation to multiple-choice questions, and handling video lengths from seconds to hours. Codes have been available at https://github.com/farewellthree/PPLLaVA.

Figures (8)

• Figure 1: (a) An instance from VideoMME fu2024video. The crucial information pertains to only a small portion of the video for different questions. (b) Performance comparison of PPLLaVA with recent strong Video LLM among video benchmarks, image benchmarks, and efficiency. All the models are based on Vicuna-7B.
• Figure 2: The overview of PPLLaVA for compressing the video based on user prompts and generating responses on the input video and instructions.
• Figure 3: Spatial pooling effects. We set $T=16$ and $k_t = d_t = 1$, varying the spatial kernel size and stride.
• Figure 4: Temporal pooling effects. We set $T=32$ and $k_w = d_w = k_h = d_h = 3$, varying the temporal kernel size and stride.
• Figure 5: The visualization of the attention weights used to guide video pooling.