Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection

Furkan Mumcu, Michael J. Jones, Anoop Cherian, Yasin Yilmaz

TL;DR

This work tackles semi-supervised video anomaly detection in a single scene by leveraging multimodal large language models to generate textual descriptions of object activities and interactions. By building a normality model from MLLM-derived descriptions of nominal data and detecting anomalies through language-based similarity to exemplar descriptions, the method provides natural explanations for detected anomalies. The approach explicitly models object interactions and supports integration with existing VAD methods, achieving state-of-the-art performance on ComplexVAD and improvements on Avenue and Street Scene when fused with tracklet-based methods. These results demonstrate not only strong detection accuracy but also interpretable, high-level reasoning about scene dynamics, suggesting practical potential for safety-critical surveillance applications.

Abstract

Existing semi-supervised video anomaly detection (VAD) methods often struggle with detecting complex anomalies involving object interactions and generally lack explainability. To overcome these limitations, we propose a novel VAD framework leveraging Multimodal Large Language Models (MLLMs). Unlike previous MLLM-based approaches that make direct anomaly judgments at the frame level, our method focuses on extracting and interpreting object activity and interactions over time. By querying an MLLM with visual inputs of object pairs at different moments, we generate textual descriptions of the activity and interactions from nominal videos. These textual descriptions serve as a high-level representation of the activity and interactions of objects in a video. They are used to detect anomalies during test time by comparing them to textual descriptions found in nominal training videos. Our approach inherently provides explainability and can be combined with many traditional VAD methods to further enhance their interpretability. Extensive experiments on benchmark datasets demonstrate that our method not only detects complex interaction-based anomalies effectively but also achieves state-of-the-art performance on datasets without interaction anomalies.

Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection

TL;DR

This work tackles semi-supervised video anomaly detection in a single scene by leveraging multimodal large language models to generate textual descriptions of object activities and interactions. By building a normality model from MLLM-derived descriptions of nominal data and detecting anomalies through language-based similarity to exemplar descriptions, the method provides natural explanations for detected anomalies. The approach explicitly models object interactions and supports integration with existing VAD methods, achieving state-of-the-art performance on ComplexVAD and improvements on Avenue and Street Scene when fused with tracklet-based methods. These results demonstrate not only strong detection accuracy but also interpretable, high-level reasoning about scene dynamics, suggesting practical potential for safety-critical surveillance applications.

Abstract

Existing semi-supervised video anomaly detection (VAD) methods often struggle with detecting complex anomalies involving object interactions and generally lack explainability. To overcome these limitations, we propose a novel VAD framework leveraging Multimodal Large Language Models (MLLMs). Unlike previous MLLM-based approaches that make direct anomaly judgments at the frame level, our method focuses on extracting and interpreting object activity and interactions over time. By querying an MLLM with visual inputs of object pairs at different moments, we generate textual descriptions of the activity and interactions from nominal videos. These textual descriptions serve as a high-level representation of the activity and interactions of objects in a video. They are used to detect anomalies during test time by comparing them to textual descriptions found in nominal training videos. Our approach inherently provides explainability and can be combined with many traditional VAD methods to further enhance their interpretability. Extensive experiments on benchmark datasets demonstrate that our method not only detects complex interaction-based anomalies effectively but also achieves state-of-the-art performance on datasets without interaction anomalies.
Paper Structure (18 sections, 8 equations, 3 figures, 6 tables)

This paper contains 18 sections, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MLLMs in Video Anomaly Detection
  4. Our Approach
  5. Object Detection and Tracking
  6. Generating Textual Descriptions
  7. Model Building
  8. Anomaly Detection in Test Video
  9. Combination with Previous Methods
  10. Experiments
  11. Quantitative Results
  12. Qualitative Results: Explainability
  13. Ablation Study
  14. Limitations and Future Directions
  15. Conclusions
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The pipeline of our method (MLLM-EVAD) for frame to textual description generation with the help of an object detector, object tracker and MLLM. Please see the text for more explanation.
  • Figure 2: Examples of anomalies in the ComplexVAD dataset that are detected by our method. In each box, the top row shows two frames of the anomalous video clip followed by the textual description generated by Gemma 3. The second row shows two frames from the closest matching exemplar to the anomaly. The textual description for the closest exemplar is also shown. By contrasting the anomalous description with the closest normal description, it is clear why the testing video clip was found to be anomalous.
  • Figure S1: The activity descriptions generated by Gemma 3 and GPT-4o with given images.