Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection

TL;DR

This work proposes Image-Event Fusion for Video Anomaly Detection (IEF-VAD), a framework that synthesizes event representations directly from RGB videos and fuses them with image features through a principled, uncertainty-aware process, enabling accurate and robust video understanding across diverse applications without requiring dedicated event sensors.

Abstract

Most existing video anomaly detectors rely solely on RGB frames, which lack the temporal resolution needed to capture abrupt or transient motion cues, key indicators of anomalous events. To address this limitation, we propose Image-Event Fusion for Video Anomaly Detection (IEF-VAD), a framework that synthesizes event representations directly from RGB videos and fuses them with image features through a principled, uncertainty-aware process. The system (i) models heavy-tailed sensor noise with a Student`s-t likelihood, deriving value-level inverse-variance weights via a Laplace approximation; (ii) applies Kalman-style frame-wise updates to balance modalities over time; and (iii) iteratively refines the fused latent state to erase residual cross-modal noise. Without any dedicated event sensor or frame-level labels, IEF-VAD sets a new state of the art across multiple real-world anomaly detection benchmarks. These findings highlight the utility of synthetic event representations in emphasizing motion cues that are often underrepresented in RGB frames, enabling accurate and robust video understanding across diverse applications without requiring dedicated event sensors. Code and models are available at https://github.com/EavnJeong/IEF-VAD.

Figures (5)

• Figure 1: Overview of IEF-VAD framework. Each video frame and its corresponding synthetic event representation are processed by CLIP encoders to obtain feature embeddings $z_m$. These are further encoded by modality-specific transformers $f_m$ to produce $\hat{z}_m$, which are then passed through projection heads $g_m$ and $h_m$ to estimate $\mu_m$ and $\sigma_m$. The estimated $\sigma_m$ is used to compute the uncertainty-aware fusion weight $w_m$, which is used to obtain the initial fused representation $\mu_f^{t, 0}$. This is refined over $N$ iterative steps through a refinement network to produce the final output $\mu_f^{t, N}$.
• Figure 2: Radar charts showing per-class anomaly detection performance (AUC and AP) for image-only (blue), event-only (orange), and fused (green) approaches. Each radial axis represents an anomaly category, and values are normalized per class by the maximum score. The fused approach consistently covers a larger area, highlighting improved detection across anomaly types.
• Figure 3: Change in image-side uncertainty weights $\Delta w_x$ under modality-specific masking perturbations across four datasets. The horizontal axis $i$ denotes the latent dimension index, and the vertical axis $\rho$ indicates the proportion of values in $z_x$ or $z_e$ that are masked to zero. Each surface visualizes $\Delta w_x(i, \rho) = w_x^{\text{masked}}(i) - w_x^{\text{clean}}(i)$ for a given masking ratio. The top row corresponds to masking applied to $z_x$ (image modality), while the bottom row applies masking to $z_e$ (event modality), with both measuring the resulting change in $w_x$. Positive values (blue) indicate increased confidence in the image modality under corruption, while negative values (red) reflect a reduction. The non-uniform patterns across $i$ highlight dimension-specific responses to value-level degradation.
• Figure :
• Figure :

Theorems & Definitions (2)

• Proposition 1: Robustness of Student's t to Outliers
• Proposition 2: Effective Variance under the Laplace Approximation