VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs

TL;DR

VANE-Bench tackles the need to evaluate Video-LMMs on video anomaly detection by compiling a diverse set of real-world and AI-generated anomalies and reframing VAD as a MC-Video QA task. It introduces a three-module semi-automatic data construction pipeline (Frame Annotation Module, Caption Generation Module, and Question Answer Generation Module) to produce 325 clips and 559 QA pairs, enabling rigorous evaluation of both open- and closed-source Video-LMMs. Across nine contemporary models, the benchmark reveals substantial gaps in detecting subtle AI-generated anomalies, though closed-source systems generally outperform open-source ones and sometimes approach human-level performance on challenging cases. The work emphasizes temporal reasoning and robust alignment as key bottlenecks and provides open-source code and data to accelerate progress in video anomaly detection for misinformation and deepfake mitigation.

Abstract

The recent developments in Large Multi-modal Video Models (Video-LMMs) have significantly enhanced our ability to interpret and analyze video data. Despite their impressive capabilities, current Video-LMMs have not been evaluated for anomaly detection tasks, which is critical to their deployment in practical scenarios e.g., towards identifying deepfakes, manipulated video content, traffic accidents and crimes. In this paper, we introduce VANE-Bench, a benchmark designed to assess the proficiency of Video-LMMs in detecting and localizing anomalies and inconsistencies in videos. Our dataset comprises an array of videos synthetically generated using existing state-of-the-art text-to-video generation models, encompassing a variety of subtle anomalies and inconsistencies grouped into five categories: unnatural transformations, unnatural appearance, pass-through, disappearance and sudden appearance. Additionally, our benchmark features real-world samples from existing anomaly detection datasets, focusing on crime-related irregularities, atypical pedestrian behavior, and unusual events. The task is structured as a visual question-answering challenge to gauge the models' ability to accurately detect and localize the anomalies within the videos. We evaluate nine existing Video-LMMs, both open and closed sources, on this benchmarking task and find that most of the models encounter difficulties in effectively identifying the subtle anomalies. In conclusion, our research offers significant insights into the current capabilities of Video-LMMs in the realm of anomaly detection, highlighting the importance of our work in evaluating and improving these models for real-world applications. Our code and data is available at https://hananshafi.github.io/vane-benchmark/

Figures (10)

• Figure 1: Samples showing the AI-Generated video category of VANE-Bench. We collect these synthetic videos from SORA sora, Open-Sora open-sora, Runway Gen2 gen2-runway, ModelScopeT2V wang2023modelscope, and VideoLCM wang2023videolcm. The correct option in each question is highlighted in bold. Note that many of these anomalies are extremely subtle and difficult for humans to detect since the changes happen in rapid succession, with the entire video played in under a second. Anomalies are identified with red bounding boxes for clarity. Note that our actual dataset does not contain bounding box overlays.
• Figure 2: Left: Performance of Video-LMMs on five anomaly categories of SORA dataset. Right: Overall performance of Video-LMMs averaged across all the benchmark datasets, including AI-generated and real-world anomaly datasets.
• Figure 3: VANE-Bench dataset statistics: Left and Middle: Composition and type of anomalies present in AI-generated and real-world videos. Right: Number of samples and QA pairs present in each type of video dataset.
• Figure 4: Flow diagram showing the semi-automatic construction process of our VANE-Bench dataset. The entire process can be divided into 3 interconnected stages/modules, i.e., i. Frame Annotation Module (FAM), ii. Caption Generation Module (CGM), iii. Question Answer Generation Module (QAGM).
• Figure 5: Human vs Video-LMMs' performance on SORA. Performance comparison of humans vs Video-LMMs on VQA task of detecting anomalies in SORA dataset. We find that closed-source Video-LMMs perform comparably to humans while open-source Video-LMMs struggle to detect subtle anomalies.