ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models

TL;DR

ViLMA introduces a zero-shot, foil-based benchmark to probe temporal visuo-linguistic grounding in video-language models. By coupling proficiency tests with main temporal reasoning tasks and rigorous human validation, it exposes weaknesses in current VidLMs' temporal understanding and possible reliance on dataset biases. The study analyzes a broad spectrum of VidLMs, image-language models, and unimodal baselines, finding that VidLMs do not markedly outperform image-based or text-only systems on temporal grounding once proficiency controls are accounted for. The results underscore the need for temporally robust, bias-resistant evaluation methods to drive future improvements in video-language grounding capabilities.

Abstract

With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as well as their performance gaps compared to human-level understanding. ViLMA also includes proficiency tests, which assess basic capabilities deemed essential to solving the main counterfactual tests. We show that current VidLMs' grounding abilities are no better than those of vision-language models which use static images. This is especially striking once the performance on proficiency tests is factored in. Our benchmark serves as a catalyst for future research on VidLMs, helping to highlight areas that still need to be explored.

Figures (22)

• Figure 1: An overview of ViLMA. A proficiency test first evaluates basic understanding skills of a model, followed by a more complex main test for a specific temporal reasoning capability.
• Figure 2: Form used in the human validation. The general instructions on the left-hand side are always visible to the annotator.
• Figure 3: Caption and foil distribution of Action Counting test, before and after Amazon Mechanical Turk validation process.
• Figure 4: Caption and foil distribution of Situation Awareness main test, before and after Amazon Mechanical Turk validation process.
• Figure 5: Caption and foil distribution of Rare Actions test, before and after Amazon Mechanical Turk validation process.