MLE-UVAD: Minimal Latent Entropy Autoencoder for Fully Unsupervised Video Anomaly Detection

Abstract

In this paper, we address the challenging problem of single-scene, fully unsupervised video anomaly detection (VAD), where raw videos containing both normal and abnormal events are used directly for training and testing without any labels. This differs sharply from prior work that either requires extensive labeling (fully or weakly supervised) or depends on normal-only videos (one-class classification), which are vulnerable to distribution shifts and contamination. We propose an entropy-guided autoencoder that detects anomalies through reconstruction error by reconstructing normal frames well while making anomalies reconstruct poorly. The key idea is to combine the standard reconstruction loss with a novel Minimal Latent Entropy (MLE) loss in the autoencoder. Reconstruction loss alone maps normal and abnormal inputs to distinct latent clusters due to their inherent differences, but also risks reconstructing anomalies too well to detect. Therefore, MLE loss addresses this by minimizing the entropy of latent embeddings, encouraging them to concentrate around high-density regions. Since normal frames dominate the raw video, sparse anomalous embeddings are pulled into the normal cluster, so the decoder emphasizes normal patterns and produces poor reconstructions for anomalies. This dual-loss design produces a clear reconstruction gap that enables effective anomaly detection. Extensive experiments on two widely used benchmarks and a challenging self-collected driving dataset demonstrate that our method achieves robust and superior performance over baselines.

Figures (8)

• Figure 1: Collapse of the anomalous latent distribution. Green: normal frames. Red: abnormal frames. Solid lines: latent distributions of the original inputs. Dashed lines: latent distributions of the reconstructions. The key idea is to pull sparse anomalous embeddings toward the dense normal embeddings, so normal frames remain well reconstructed while anomalies reconstruct poorly.
• Figure 2: MLE-UVAD pipeline: unsupervised training and detection through reconstruction gap. Unlabeled videos are directly put into an autoencoder trained with a dual loss: (1) The MSE loss reconstructs all frames accurately. (2) The MLE collapses sparse abnormal embeddings toward the dense normal embedding, making anomaly reconstructions worse relative to normals. An anomaly is detected by setting a threshold on the reconstruction error.
• Figure 3: t-SNE visualization of latent embeddings across different methods (blue = normal, orange = abnormal). Baselines, only MSE and GCL Zaheer_GCL, keep the normal and abnormal embeddings separate for better reconstruction. In contrast, our proposed MLE loss regularizes the latent distribution by collapsing the abnormal distribution into the dominant normal cluster.
• Figure 4: Unsupervised anomaly detection performance across three benchmarks. Top row: baselines (TMAE, GCL, Vanilla CAE). Bottom row: our MLE-Guided CAE. Columns: DonkeyCar, UBnormal, Corridor. Our method shows a clear normal–anomaly separation of the PCC value.
• Figure 5: Effect of kernel size $\sigma$ in the MLE loss. Left: PCC trajectories for different $\sigma$ values; shaded gray regions indicate ground-truth anomalies. Right: AUC heatmap across epochs. Mid-range $\sigma$ (0.01–0.1) yields stable reconstructions and high AUC, while very small or large $\sigma$ degrades detection.