Instance-Aligned Captions for Explainable Video Anomaly Detection

TL;DR

This paper tackles the lack of verifiable explanations in video anomaly detection by introducing instance-aligned captions that tie each textual claim to specific object instances with appearance and motion attributes. The authors implement a four-stage annotation pipeline and apply it to eight VAD benchmarks, expanding VIEW360 to VIEW360+ with richer spatial and anomaly diversity. They define a grounded evaluation protocol comprising caption quality, spatial grounding, a joint segmentation-caption score, and false-positive entity counts, and demonstrate that current LLM/VLM-based explanations struggle with grounding and consistency. The work establishes a new benchmark and dataset resources that reveal critical failure modes in existing methods and provide a foundation for developing trustworthy, interpretable VAD systems, especially for assistive 360° egocentric scenarios.

Abstract

Explainable video anomaly detection (VAD) is crucial for safety-critical applications, yet even with recent progress, much of the research still lacks spatial grounding, making the explanations unverifiable. This limitation is especially pronounced in multi-entity interactions, where existing explainable VAD methods often produce incomplete or visually misaligned descriptions, reducing their trustworthiness. To address these challenges, we introduce instance-aligned captions that link each textual claim to specific object instances with appearance and motion attributes. Our framework captures who caused the anomaly, what each entity was doing, whom it affected, and where the explanationis grounded, enabling verifiable and actionable reasoning. We annotate eight widely used VAD benchmarks and extend the 360-degree egocentric dataset, VIEW360, with 868 additional videos, eight locations, and four new anomaly types, creating VIEW360+, a comprehensive testbed for explainable VAD. Experiments show that our instance-level spatially grounded captions reveal significant limitations in current LLM- and VLM-based methods while providing a robust benchmark for future research in trustworthy and interpretable anomaly detection.

Figures (11)

• Figure 1: Comparison of anomaly understanding paradigms. (a) Traditional score-only detection raises an alert but provides no explainability. (b) LLM/VLM-based systems generate textual explanations but lack spatial grounding—when multiple people match the description or the model attends to wrong objects, explanations become ambiguous and unverifiable. (c) Our approach generates multiple captions, one for each object instance, and links them to instance segmentation masks. Each caption explicitly separates appearance (App.) and motion (Motion) attributes, capturing all participants and their actions to enable complete and verifiable scene understanding.
• Figure 2: Comparison of anomaly‐understanding paradigms. (a) Traditional VAD predicts only anomaly scores without explanations. (b) VLM‐based VAD generates textual descriptions but lacks object‐level grounding. (c) Grounding VLMs provide spatial localization but do not produce object‐specific explanations. (d) Our instance-aligned captioning pipeline links each detected entity to its grounded instance track and generates captions for all relevant participants, enabling complete who–what–whom–where reasoning for anomaly events.
• Figure 3: Our four-stage annotation pipeline for generating instance-aligned caption. Starting from video clips, we extract reference captions, manually annotate objects with bounding boxes and role labels, track them through SAM2 to produce segmentation masks, and finally generate object-specific captions by combining cropped object sequences with reference context.
• Figure 4: Examples of abnormal events in VIEW360 and the expanded VIEW360+. VIEW360+ introduces new anomaly types, additional locations, and full instance-level annotations with aligned captions, enabling richer $360^{\circ}$ anomaly understanding.
• Figure 5: Visualization of explanation quality on a UCF-Crime anomaly scenario