PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understanding

TL;DR

This paper addresses the need for interpretable, real-time video anomaly understanding without the heavy cost of fine-tuning multimodal LLMs or relying on external captioning modules. It introduces PrismVAU, a two-stage system that combines coarse anchor-based anomaly scoring with MLLM-based refinement, guided by weakly supervised Automatic Prompt Engineering to optimize both anchors and prompts. The approach achieves competitive performance among non-fine-tuned VAU methods on UCF-Crime and XD-Violence while providing semantic explanations, with real-time inference and low computational overhead. The work demonstrates the complementary strengths of anchor-based scoring and multimodal reasoning, and suggests extensions to broader event categories, cross-dataset generalization, and standardized evaluation of explanations for transparent VAU systems.

Abstract

Video Anomaly Understanding (VAU) extends traditional Video Anomaly Detection (VAD) by not only localizing anomalies but also describing and reasoning about their context. Existing VAU approaches often rely on fine-tuned multimodal large language models (MLLMs) or external modules such as video captioners, which introduce costly annotations, complex training pipelines, and high inference overhead. In this work, we introduce PrismVAU, a lightweight yet effective system for real-time VAU that leverages a single off-the-shelf MLLM for anomaly scoring, explanation, and prompt optimization. PrismVAU operates in two complementary stages: (1) a coarse anomaly scoring module that computes frame-level anomaly scores via similarity to textual anchors, and (2) an MLLM-based refinement module that contextualizes anomalies through system and user prompts. Both textual anchors and prompts are optimized with a weakly supervised Automatic Prompt Engineering (APE) framework. Extensive experiments on standard VAD benchmarks demonstrate that PrismVAU delivers competitive detection performance and interpretable anomaly explanations -- without relying on instruction tuning, frame-level annotations, and external modules or dense processing -- making it an efficient and practical solution for real-world applications.

Figures (6)

• Figure 1: PrismVAU overview. Gray denotes inputs, blue indicates off-the-shelf MLLM modules, and orange highlights our contributions. A coarse anomaly curve is computed from the video segment and textual anchor embeddings, then injected ("CI’’) into the MLLM prompt. The MLLM outputs a textual explanation, a segment-level anomaly score, and an anomalous temporal region, which are used to refine the initial curve into a temporally precise and semantically informed anomaly signal.
• Figure 2: Diagram of the proposed APE framework for optimizing both textual anchors and MLLM prompts. The same MLLM is used for inference and optimization.
• Figure 3: Comparison of initial normal/abnormal textual anchors (a) with their optimized counterparts (b).
• Figure 4: Learning curves produced by the APE optimization of textual anchors (a) and of the MLLM system and user prompts (b) on UCF-Crime.
• Figure 5: Results on Explosion033_x264 from UCF-Crime. Ground-truth (GT) anomalous intervals are compared against the coarse anomaly curve from the textual anchors (TA), the step-like MLLM predictions, and the refined anomaly curve. Human-readable explanations generated by the MLLM are shown as speech bubbles. Bold ticks on the x-axis indicate video segment partitions used when processing the untrimmed video. Sample frames on the right illustrate key anomaly events.