Just Dance with $π$! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection

TL;DR

PI-VAD tackles the limitations of RGB-only weakly supervised video anomaly detection by leveraging five additional modalities during training. It introduces a Poly-modal Inductor comprising a Pseudo Modality Generation module and a Cross Modal Induction module to synthesize and align modality embeddings with RGB, enabling a robust multi-modal representation while keeping inference light. The approach demonstrates state-of-the-art performance on UCF-Crime, XD-Violence, and MSAD datasets and runs in real-time at 30 FPS. The work offers a scalable framework for incorporating additional modalities without runtime backbones, broadening applicability to real-world surveillance and safety tasks.

Abstract

Weakly-supervised methods for video anomaly detection (VAD) are conventionally based merely on RGB spatio-temporal features, which continues to limit their reliability in real-world scenarios. This is due to the fact that RGB-features are not sufficiently distinctive in setting apart categories such as shoplifting from visually similar events. Therefore, towards robust complex real-world VAD, it is essential to augment RGB spatio-temporal features by additional modalities. Motivated by this, we introduce the Poly-modal Induced framework for VAD: "PI-VAD", a novel approach that augments RGB representations by five additional modalities. Specifically, the modalities include sensitivity to fine-grained motion (Pose), three dimensional scene and entity representation (Depth), surrounding objects (Panoptic masks), global motion (optical flow), as well as language cues (VLM). Each modality represents an axis of a polygon, streamlined to add salient cues to RGB. PI-VAD includes two plug-in modules, namely Pseudo-modality Generation module and Cross Modal Induction module, which generate modality-specific prototypical representation and, thereby, induce multi-modal information into RGB cues. These modules operate by performing anomaly-aware auxiliary tasks and necessitate five modality backbones -- only during training. Notably, PI-VAD achieves state-of-the-art accuracy on three prominent VAD datasets encompassing real-world scenarios, without requiring the computational overhead of five modality backbones at inference.

Figures (6)

• Figure 1: a): Illustration of abnormal frames and respective multi-modal saliencies in complex real-world scenes. Optical flow captures distinct abnormal motion in "Abuse” and "Arrest”, while depth and pose detect subtle movements that optical flow may miss. Panoptic masks and text provide overall scene context. b): Comparison of multi-modal methods with our PI-VAD. PI-VAD requires the five modalities only during training, significantly reducing computation and enabling real-world applicability.
• Figure 2: (a) Overview of Poly-modal Induced VAD ($\pi$-VAD): In the training phase, $\pi$-VAD uses a teacher-student approach, where a poly-modal inductor enhances the student's RGB representation by generating and associating five distinct modalities. Note that the teacher’s weights remain fixed during training. At inference, the student and poly-modal inductor operate independently to detect video anomalies. (b) Poly-modal Inductor (PI): PI refines the student’s intermediate feature, $\mathcal{F}^*$, by generating pseudo-modalities through a modality generation module (PMG). These generated modalities are then combined with $\mathcal{F}^*$ to produce an enhanced feature set, $\mathcal{F}^*_M$.
• Figure 3: Class-wise $AUC$ comparison of $\pi$-VAD with UR-DMU AAAI23URDMU on the UCF-Crime dataset.
• Figure 4: Visualization of sample frames and ground truth (green shed) vs. prediction scores (red shed) for various cases in Row-1 and Row-2. For each plot in Row-2, the X and Y axis denotes the number of frames and corresponding anomaly scores. Row-3 shows the latent activation learned by multi-modality. We plot the mean value of the normalized modalities activations from the first transformer block of the late PI module to show the alignment between modalities and their correlation to the predicted abnormal scores.
• Figure 5: Class-wise $AUC$ comparison between the RGB model and RGB with one additional modality model on UCF-Crime.