Multimodal Real-Time Anomaly Detection and Industrial Applications

TL;DR

The paper tackles real-time anomaly detection in industrial environments by fusing synchronized video and audio. It documents the evolution from a lightweight baseline (YOLOv8, ByteTrack, AST) to an advanced system with a multi-model audio ensemble (AST, Wav2Vec2, HuBERT), a hybrid detector (YOLO+DETR), and bidirectional cross-modal attention. It introduces multi-method anomaly detection and industrial monitoring enhancements, demonstrating real-time performance on standard hardware. The work shows practical implications for safety-critical applications through robust multimodal fusion and comprehensive evaluation across diverse industrial scenarios.

Abstract

This paper presents the design, implementation, and evolution of a comprehensive multimodal room-monitoring system that integrates synchronized video and audio processing for real-time activity recognition and anomaly detection. We describe two iterations of the system: an initial lightweight implementation using YOLOv8, ByteTrack, and the Audio Spectrogram Transformer (AST), and an advanced version that incorporates multi-model audio ensembles, hybrid object detection, bidirectional cross-modal attention, and multi-method anomaly detection. The evolution demonstrates significant improvements in accuracy, robustness, and industrial applicability. The advanced system combines three audio models (AST, Wav2Vec2, and HuBERT) for comprehensive audio understanding, dual object detectors (YOLO and DETR) for improved accuracy, and sophisticated fusion mechanisms for enhanced cross-modal learning. Experimental evaluation shows the system's effectiveness in general monitoring scenarios as well as specialized industrial safety applications, achieving real-time performance on standard hardware while maintaining high accuracy.

Figures (11)

• Figure 1: Initial system architecture showing the four main layers: sensor, preprocessing, model, and fusion layers. This lightweight design prioritizes real-time performance and computational efficiency.
• Figure 2: Advanced system architecture incorporating multi-model audio ensemble, hybrid object detection, bidirectional cross-modal attention, and multi-method anomaly detection. The system maintains real-time performance while significantly improving accuracy and robustness.
• Figure 3: For the Case of Finger snapping the green vector flows tell how the objects are moving
• Figure 4: Audio classification performance Multi-model ensemble (AST+Wav2Vec2+HuBERT) showing improved accuracy and robustness.
• Figure 5: Industrial monitoring results we tried to create a fire which was marked as anomaly and its image was also saved in the anomalies section