Advancing Video Anomaly Detection: A Bi-Directional Hybrid Framework for Enhanced Single- and Multi-Task Approaches

TL;DR

The paper tackles video anomaly detection by arguing that optimizing a single strong proxy task can uplift both single-task and multi-task VAD. It presents a bi-directional hybrid framework that uses middle-frame prediction, a convolutional temporal transformer backbone, and a layer-interactive ConvLSTM bridge to fuse global temporal context with fine-grained local details. The approach achieves state-of-the-art or competitive frame-level AUC on UCSD Ped2, Avenue, ShanghaiTech, and Street Scene, while maintaining a compact model and fast inference without GANs. The work further shows that combining vision transformers and ConvLSTMs yields robust anomaly localization and stability across diverse scenes, reinforcing the value of task-specific framework design in VAD.

Abstract

Despite the prevailing transition from single-task to multi-task approaches in video anomaly detection, we observe that many adopt sub-optimal frameworks for individual proxy tasks. Motivated by this, we contend that optimizing single-task frameworks can advance both single- and multi-task approaches. Accordingly, we leverage middle-frame prediction as the primary proxy task, and introduce an effective hybrid framework designed to generate accurate predictions for normal frames and flawed predictions for abnormal frames. This hybrid framework is built upon a bi-directional structure that seamlessly integrates both vision transformers and ConvLSTMs. Specifically, we utilize this bi-directional structure to fully analyze the temporal dimension by predicting frames in both forward and backward directions, significantly boosting the detection stability. Given the transformer's capacity to model long-range contextual dependencies, we develop a convolutional temporal transformer that efficiently associates feature maps from all context frames to generate attention-based predictions for target frames. Furthermore, we devise a layer-interactive ConvLSTM bridge that facilitates the smooth flow of low-level features across layers and time-steps, thereby strengthening predictions with fine details. Anomalies are eventually identified by scrutinizing the discrepancies between target frames and their corresponding predictions. Several experiments conducted on public benchmarks affirm the efficacy of our hybrid framework, whether used as a standalone single-task approach or integrated as a branch in a multi-task approach. These experiments also underscore the advantages of merging vision transformers and ConvLSTMs for video anomaly detection.

Figures (11)

• Figure 1: Overview of the proposed framework. It includes an encoding pipeline and two decoding pipeline (forward and backward). The encoding pipeline consists of a spatial encoder and a Convolutional Temporal Transformer (ConvTTrans) encoder. Either the forward or backward decoding pipeline comprises the same spatial encoder from the encoding pipeline, a ConvTTrans decoder, a layer-interactive ConvLSTM bridge, and a spatial decoder.
• Figure 2: The encoder-decoder architecture of the Convolutional Temporal Transformer (ConvTTrans). Only the encoder and forward decoder are visualized; the backward decoder behaves similarly to the forward decoder but with different inputs.
• Figure 3: Illustration of the multi-head ConvTSA. The attention-based representation $\mathcal{H}_{i}$ between frame $\mathcal{X}_{i}$ (embedded into $\mathcal{E}_{i}$) and every other frame $\mathcal{X}_{j}$ in the clip (embedded into $\mathcal{E}_{j}$) is simultaneously calculated in multiple heads. Note that the actual number of $\{\mathcal{E}_j\}$ depends on the clip length.
• Figure 4: Architecture of the layer-interactive ConvLSTM bridge and its connection with the spatial encoder and decoder. ConvLSTM $\alpha$ and ConvLSTM $\beta$ are the traditional and modified ConvLSTMs, respectively; $H^{\alpha}_{i}$ and orange dashed lines indicate the inter-layer transition of hidden state from ConvLSTM $\alpha$ to ConvLSTM $\beta$; and grey dashed lines symbolize the standard temporal transition of hidden and cell states within ConvLSTMs shi2015convolutional. Each module corresponds to the module depicted in the same color in Fig. \ref{['fig:modeloverview']}.
• Figure 5: Pixel-level anomaly detection instances on UCSD mahadevan2010anomaly, Avenue lu2013abnormal, SHTech luo2017revisit, and StreetScene ramachandra2020street datasets, presented from top to bottom. For each dataset, the first row displays the ground-truth frames, with ground-truth anomalies highlighted in yellow boxes. The second row exhibits the prediction error maps generated by our framework, where brighter colors indicate larger prediction errors.