Driver Assistance System Based on Multimodal Data Hazard Detection
Long Zhouxiang, Ovanes Petrosian
TL;DR
This work tackles the challenge of detecting driving hazards under long-tailed event distributions by proposing an end-to-end multimodal driver-assistance system that fuses road condition video, driver facial video, and audio using an attention-based intermediate fusion mechanism. The approach leverages three dedicated branches (audio, driver video, road video) with EfficientFace-inspired feature extractors and MFCC features for audio, integrating temporal information via 1D convolutions and a pairwise attention scheme $A_{ij} = softmax((Φ_i W_q W_k^T Φ_j^T)/sqrt(d))$ to capture cross-modal correlations. A new three-modality dataset is created in simulation (AirSimNH with Need-for-Speed content) and annotated for safe/dangerous driving, enabling end-to-end multimodal learning. Experimental results on this dataset show the tri-modal model achieves 96.875% accuracy in dangerous driving state recognition, outperforming unimodal and bimodal baselines and demonstrating improved robustness and cross-modal synergy, with potential impact on real-time driving safety and ADAS development.
Abstract
Autonomous driving technology has advanced significantly, yet detecting driving anomalies remains a major challenge due to the long-tailed distribution of driving events. Existing methods primarily rely on single-modal road condition video data, which limits their ability to capture rare and unpredictable driving incidents. This paper proposes a multimodal driver assistance detection system that integrates road condition video, driver facial video, and audio data to enhance incident recognition accuracy. Our model employs an attention-based intermediate fusion strategy, enabling end-to-end learning without separate feature extraction. To support this approach, we develop a new three-modality dataset using a driving simulator. Experimental results demonstrate that our method effectively captures cross-modal correlations, reducing misjudgments and improving driving safety.