FineVAU: A Novel Human-Aligned Benchmark for Fine-Grained Video Anomaly Understanding

TL;DR

FineVAU addresses the lack of human-aligned evaluation in Video Anomaly Understanding (VAU) by framing VAU as a three-dimensional task (What, Who, Where) and presenting FV-Score, an LVLM-driven metric, alongside FineW3, a fully automated, fine-grained dataset. The approach formalizes ground truth as $G=G_{what}\cup G_{who}\cup G_{where}$ and optimizes a composite score $S(R)=\lambda_{what}\mathcal{J}_{what}(R)+\lambda_{who}\mathcal{J}_{who}(R)+\lambda_{where}\mathcal{J}_{where}(R)$, using a semantic membership function $m_{\theta}$ to assess element presence. FV-Score leverages an LVLM judge (FineVAU-Judge) via a prompt (the triplet $(R,G,P)$) and a frontier model to produce per-element memberships, enabling interpretable feedback. Experiments across five LVLMs show robust grounding for static/scene information but limited capability for spatial-temporal fine-grained events, with InternVL3 achieving the best overall performance, highlighting a new benchmark frontier for targeted training and evaluation in VAU.

Abstract

Video Anomaly Understanding (VAU) is a novel task focused on describing unusual occurrences in videos. Despite growing interest, the evaluation of VAU remains an open challenge. Existing benchmarks rely on n-gram-based metrics (e.g., BLEU, ROUGE-L) or LLM-based evaluation. The first fails to capture the rich, free-form, and visually grounded nature of LVLM responses, while the latter focuses on assessing language quality over factual relevance, often resulting in subjective judgments that are misaligned with human perception. In this work, we address this issue by proposing FineVAU, a new benchmark for VAU that shifts the focus towards rich, fine-grained and domain-specific understanding of anomalous videos. We formulate VAU as a three-fold problem, with the goal of comprehensively understanding key descriptive elements of anomalies in video: events (What), participating entities (Who) and location (Where). Our benchmark introduces a) FVScore, a novel, human-aligned evaluation metric that assesses the presence of critical visual elements in LVLM answers, providing interpretable, fine-grained feedback; and b) FineW3, a novel, comprehensive dataset curated through a structured and fully automatic procedure that augments existing human annotations with high quality, fine-grained visual information. Human evaluation reveals that our proposed metric has a superior alignment with human perception of anomalies in comparison to current approaches. Detailed experiments on FineVAU unveil critical limitations in LVLM's ability to perceive anomalous events that require spatial and fine-grained temporal understanding, despite strong performance on coarse grain, static information, and events with strong visual cues.

Figures (6)

• Figure 1: Evaluation in Video Anomaly Understanding. The performance of VAU models is commonly assessed with metrics that a) disregard semantic equivalence; b) focus on language criteria such as fluency and detail; and c) yield non-interpretable, vague scores. Existing metrics fail to signal if the model provides a factually incorrect but coherent description. In contrast, our metric can correctly classify the description as incorrect by decomposing it into the key elements humans rely on to perceive anomalies.
• Figure 2: Our two-stage, fully automated pipeline for scalable annotation of fine-grained VAU data. We ground our annotation process on high-quality human annotations, and use a LVLM to: 1) augment and refine existing events and identify the entities involved; an 2) augment entity and location information with fine-grained physical attributes.
• Figure 3: Annotation count per dimension. We split events into normal and abnormal, and separate counts of entities and their physical attributes. The large number of annotations stems from the granularity of our annotations.
• Figure 4: Word Cloud for event annotations in Fine$W^{3}$. Events often describe motion (e.g., "run", "walk"), interaction ("exit", "approach") and abnormality ("fall", "damage", "struggle", "grab", "kick")
• Figure 5: Histogram of Video Duration. Our dataset contains challenging long videos with up to 1h duration. This is expected given the source of the videos of our dataset, composed of CCTV footage.