VideoAds for Fast-Paced Video Understanding

TL;DR

VideoAds introduces the first dedicated benchmark for evaluating multi-modal large language models on advertisement videos, emphasizing complex temporal reasoning and high-density visual narratives. It provides a novel video complexity measure based on visual feature variance and constructs 1,100 high-quality VQA tasks across three categories, enabling rigorous cross-model evaluation. Experimental results show open-source LMMs increasingly close the gap with proprietary systems on surface tasks, but humans still outperform by a wide margin, especially in reasoning; audio transcripts and Chain-of-Thought prompts significantly influence performance, underscoring the need for better cross-modal reasoning and long-context modeling. The dataset and accompanying code aim to propel research on temporal dynamics and narrative understanding in short, information-rich videos with high FPS sampling requirements.

Abstract

Advertisement videos serve as a rich and valuable source of purpose-driven information, encompassing high-quality visual, textual, and contextual cues designed to engage viewers. They are often more complex than general videos of similar duration due to their structured narratives and rapid scene transitions, posing significant challenges to multi-modal large language models (MLLMs). In this work, we introduce VideoAds, the first dataset tailored for benchmarking the performance of MLLMs on advertisement videos. VideoAds comprises well-curated advertisement videos with complex temporal structures, accompanied by \textbf{manually} annotated diverse questions across three core tasks: visual finding, video summary, and visual reasoning. We propose a quantitative measure to compare VideoAds against existing benchmarks in terms of video complexity. Through extensive experiments, we find that Qwen2.5-VL-72B, an opensource MLLM, achieves 73.35\% accuracy on VideoAds, outperforming GPT-4o (66.82\%) and Gemini-1.5 Pro (69.66\%); the two proprietary models especially fall behind the opensource model in video summarization and reasoning, but perform the best in visual finding. Notably, human experts easily achieve a remarkable accuracy of 94.27\%. These results underscore the necessity of advancing MLLMs' temporal modeling capabilities and highlight VideoAds as a potentially pivotal benchmark for future research in understanding video that requires high FPS sampling. The dataset and evaluation code will be publicly available at https://videoadsbenchmark.netlify.app.

Figures (4)

• Figure 1: Recent years have witnessed a significant increase in the complexity of video benchmarks, paralleling the rapid progress in the capabilities of multi-modal large language models (MLLMs). In this work, we introduce VideoAds, a complex dataset based on advertisement videos, specifically designed to benchmark the performance of MLLMs on challenging visual comprehension and complex temporal reasoning. The size of each scatter point represents the average duration of videos within each dataset.
• Figure 2: VideoAds comprises three challenging tasks: Visual Finding, Visual Summary, and Visual Reasoning, specifically designed to evaluate MLLMs' temporal reasoning capabilities on videos with complex temporal structures that have never been investigated before. Unlike many previous datasets that focus on recognizing isolated actions or events, VideoAds demands that models derive the correct answers only through multistep reasoning over multi-modal visual clues.
• Figure 3: Visualization of Video Complexity Scores: Examples of videos with high, medium, and low complexity scores based on the proposed video complexity metrics. Videos with high complexity scores exhibit frequent scene transitions, dynamic interactions, and complex visual transformations, posing significant challenges for temporal reasoning. In contrast, videos with low complexity display static or minimally changing frames, resembling image slideshows with minimal narrative progression.
• Figure 4: Impact of Chain of thought on the model's performance for challenging reasoning tasks, we can observe a variance in the model's performance along the model size.