Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cascading multi-agent anomaly detection in surveillance systems via vision-language models and embedding-based classification

Tayyab Rehman, Giovanni De Gasperis, Aly Shmahell

TL;DR

This work tackles the challenge of real-time video anomaly detection with semantic interpretability for large-scale surveillance. It proposes a cascading multi-agent architecture that unifies fast object detection (YOLOv8), reconstruction-based anomaly scoring (autoencoder), and vision-language reasoning with an embedding-based classifier, coordinated by an event-driven and a cyclical health-monitoring agent through a publish-subscribe backbone. The approach achieves substantial latency reductions while maintaining perceptual fidelity and interpretable labeling, demonstrated on the UCF-Crime dataset with 6,990 detected events across 329k frames. The framework emphasizes early exits for routine cases, adaptable semantic reasoning for ambiguous events, and practical deployment considerations, including privacy and cross-dataset robustness as future directions.

Abstract

Intelligent anomaly detection in dynamic visual environments requires reconciling real-time performance with semantic interpretability. Conventional approaches address only fragments of this challenge. Reconstruction-based models capture low-level deviations without contextual reasoning, object detectors provide speed but limited semantics, and large vision-language systems deliver interpretability at prohibitive computational cost. This work introduces a cascading multi-agent framework that unifies these complementary paradigms into a coherent and interpretable architecture. Early modules perform reconstruction-gated filtering and object-level assessment, while higher-level reasoning agents are selectively invoked to interpret semantically ambiguous events. The system employs adaptive escalation thresholds and a publish-subscribe communication backbone, enabling asynchronous coordination and scalable deployment across heterogeneous hardware. Extensive evaluation on large-scale monitoring data demonstrates that the proposed cascade achieves a threefold reduction in latency compared to direct vision-language inference, while maintaining high perceptual fidelity (PSNR = 38.3 dB, SSIM = 0.965) and consistent semantic labeling. The framework advances beyond conventional detection pipelines by combining early-exit efficiency, adaptive multi-agent reasoning, and explainable anomaly attribution, establishing a reproducible and energy-efficient foundation for scalable intelligent visual monitoring.

Cascading multi-agent anomaly detection in surveillance systems via vision-language models and embedding-based classification

TL;DR

This work tackles the challenge of real-time video anomaly detection with semantic interpretability for large-scale surveillance. It proposes a cascading multi-agent architecture that unifies fast object detection (YOLOv8), reconstruction-based anomaly scoring (autoencoder), and vision-language reasoning with an embedding-based classifier, coordinated by an event-driven and a cyclical health-monitoring agent through a publish-subscribe backbone. The approach achieves substantial latency reductions while maintaining perceptual fidelity and interpretable labeling, demonstrated on the UCF-Crime dataset with 6,990 detected events across 329k frames. The framework emphasizes early exits for routine cases, adaptable semantic reasoning for ambiguous events, and practical deployment considerations, including privacy and cross-dataset robustness as future directions.

Abstract

Intelligent anomaly detection in dynamic visual environments requires reconciling real-time performance with semantic interpretability. Conventional approaches address only fragments of this challenge. Reconstruction-based models capture low-level deviations without contextual reasoning, object detectors provide speed but limited semantics, and large vision-language systems deliver interpretability at prohibitive computational cost. This work introduces a cascading multi-agent framework that unifies these complementary paradigms into a coherent and interpretable architecture. Early modules perform reconstruction-gated filtering and object-level assessment, while higher-level reasoning agents are selectively invoked to interpret semantically ambiguous events. The system employs adaptive escalation thresholds and a publish-subscribe communication backbone, enabling asynchronous coordination and scalable deployment across heterogeneous hardware. Extensive evaluation on large-scale monitoring data demonstrates that the proposed cascade achieves a threefold reduction in latency compared to direct vision-language inference, while maintaining high perceptual fidelity (PSNR = 38.3 dB, SSIM = 0.965) and consistent semantic labeling. The framework advances beyond conventional detection pipelines by combining early-exit efficiency, adaptive multi-agent reasoning, and explainable anomaly attribution, establishing a reproducible and energy-efficient foundation for scalable intelligent visual monitoring.
Paper Structure (31 sections, 9 equations, 7 figures, 6 tables)

This paper contains 31 sections, 9 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Introduction
  3. Related Work
  4. Proposed Methodology
  5. Multi-Agent Architecture
  6. Cascading Detection Pipeline
  7. Early-Exit Strategy
  8. Implementation Details
  9. Case Study
  10. Scenario: Camera Obstruction with Concurrent Suspicious Activity
  11. Stage I (YOLOv8 Detection).
  12. Stage II (Autoencoder Scoring).
  13. Stage III (VLM Reasoning).
  14. System-Level Response.
  15. Efficiency Gain.
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Proposed Methodology.
  • Figure 2: Training and validation MSE loss curves across 20 epochs. Loss converges within the first 10 epochs and stabilizes near $1.7 \times 10^{-4}$.
  • Figure 3: PSNR across training epochs. Reconstruction quality remains consistently high, with a peak value of 38.45 dB at epoch 17.
  • Figure 4: SSIM across training epochs. Structural fidelity remains consistently high, with the best SSIM (0.968) achieved at epoch 20.
  • Figure 5: Integrated anomaly detection dashboard over $329$k frames. The interface lists, for each merged event, the predicted type (post VLM$\to$Classifier), reconstruction error statistics, temporal duration, and source file/stream. In total, $6{,}990$ events were detected.
  • ...and 2 more figures