V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction

TL;DR

The paper introduces V2P-Bench, a visual-prompt–driven benchmark to evaluate video understanding in multimodal human-model interaction, addressing the limitations of text-based prompts in LVLM assessments. It defines 5 tasks across 12 dimensions and assembles 980 videos with 1,172 MC QA pairs drawn from 12 datasets, spanning short-to-long durations and diverse video types. Extensive experiments across 16 LVLMs (4 closed-source, 12 open-source) and human experts reveal a sizable gap between state-of-the-art models and human performance, underscoring current shortcomings in video visual-prompt understanding. The work provides a foundation for advancing multimodal interaction and prompts a push toward more robust, instance-level video understanding benchmarks.

Abstract

Large Vision-Language Models (LVLMs) have made significant progress in the field of video understanding recently. However, current benchmarks uniformly lean on text prompts for evaluation, which often necessitate complex referential language and fail to provide precise spatial and temporal references. This limitation diminishes the experience and efficiency of human-model interaction. To address this limitation, we propose the Video Visual Prompt Benchmark(V2P-Bench), a comprehensive benchmark specifically designed to evaluate LVLMs' video understanding capabilities in multimodal human-model interaction scenarios. V2P-Bench includes 980 unique videos and 1,172 QA pairs, covering 5 main tasks and 12 dimensions, facilitating instance-level fine-grained understanding aligned with human cognition. Benchmarking results reveal that even the most powerful models perform poorly on V2P-Bench (65.4% for GPT-4o and 67.9% for Gemini-1.5-Pro), significantly lower than the human experts' 88.3%, highlighting the current shortcomings of LVLMs in understanding video visual prompts. We hope V2P-Bench will serve as a foundation for advancing multimodal human-model interaction and video understanding evaluation. Project page: https://github.com/gaotiexinqu/V2P-Bench.

Figures (17)

• Figure 1: (a)(b) shows the comparisons of pure text prompting and visual prompting for video understanding. Simply overlaying visual prompts on video frames can enhance the user experience in Human-Model Interaction(HMI) while simultaneously reducing the difficulty for LVLMs to understand user intentions, particularly in complex environments where referential ambiguity is prevalent. (c) shows an example of V2P-Bench. The ground-truth answer is highlighted in green. Full video could be found at youtu.be/lDlA7cfNk8A.
• Figure 2: (Left)Datasets and categories. Our dataset is derived from 12 datasets and contains 20 restructured categories. (Right)Performance radar chart. We report the performance of different models on V2P-Bench by dimension. SOTA for each dimension is given.
• Figure 3: Distribution of QA dimensions and video durations. V2P-Bench encompasses 5 main tasks, 12 dimensions and covers a wide range of video lengths, enabling an comprehensive assessment of LVLMs.
• Figure 4: Various visual prompt types. We don't consider mask as it would significantly obscure the features of the target, even with a small alpha value (e.g. 0.2).
• Figure 5: Results on two data formats.Retrieval refers to the sequential input of the original video, questions, and visual prompt frames; Needle refers to embedding the visual prompt frames into the video.