Video Anomaly Detection with Semantics-Aware Information Bottleneck

TL;DR

This work tackles semi-supervised video anomaly detection by replacing fixed memory prototypes with an adaptive information bottleneck filtering mechanism that compresses normal features into low-dimensional manifolds. It further introduces a multimodal framework that jointly models appearance, motion, and high-level semantics through a Tri-Modal Fusion Encoder and a Multimodal Joint Attention Decoder. A novel inter-modality consistency objective, including a motion frame-difference contrastive loss, enforces cross-modal coherence and robust anomaly scoring. Empirical results on ShanghaiTech, Avenue, and Ped2 demonstrate state-of-the-art performance and strong ablations validating the effectiveness of the bottleneck filtering and multimodal design.

Abstract

Semi-supervised video anomaly detection methods face two critical challenges: (1) Strong generalization blurs the boundary between normal and abnormal patterns. Although existing approaches attempt to alleviate this issue using memory modules, their rigid prototype-matching process limits adaptability to diverse scenarios; (2) Relying solely on low-level appearance and motion cues makes it difficult to perceive high-level semantic anomalies in complex scenes. To address these limitations, we propose SIB-VAD, a novel framework based on adaptive information bottleneck filtering and semantic-aware enhancement. We propose the Sparse Feature Filtering Module (SFFM) to replace traditional memory modules. It compresses normal features directly into a low-dimensional manifold based on the information bottleneck principle and uses an adaptive routing mechanism to dynamically select the most suitable normal bottleneck subspace. Trained only on normal data, SFFMs only learn normal low-dimensional manifolds, while abnormal features deviate and are effectively filtered. Unlike memory modules, SFFM directly removes abnormal information and adaptively handles scene variations. To improve semantic awareness, we further design a multimodal prediction framework that jointly models appearance, motion, and semantics. Through multimodal consistency constraints and joint error computation, it achieves more robust VAD performance. Experimental results validate the effectiveness of our feature filtering paradigm based on semantics-aware information bottleneck. Project page at https://qzfm.github.io/sib_vad_project_page/

Figures (4)

• Figure 1: Comparison between the previous and our proposed framework. (a) Existing memory modules filter abnormal information through inflexible prototype matching. (b) Our proposed sparse feature filtering paradigm achieves better VAD performance compared to memory-based methods.
• Figure 2: Overview of SIB-VAD. Firstly, TMFE (Sec. \ref{['tmfe']}) extracts descriptions from the input clips and encodes and fuses semantic, appearance, and motion features. Secondly, MJAD (Sec. ref mmjd) performs joint decoding of the fused features to predict the next frame and semantic features. SFFM (Sec. \ref{['sffm']}) filters abnormal information to increase the prediction error when anomalies occur. Finally, anomaly scores are calculated based on frame prediction errors and semantic errors (Sec. \ref{['anomaly_detection']}).
• Figure 3: Visualization of anomaly score curves. The images above show the corresponding abnormal or normal events.
• Figure 4: Visualization of frame-centric predictions and error maps. In the error maps, brighter areas indicate higher levels of abnormality in those regions.