IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengers

TL;DR

The IC3M is introduced, an efficient camera-rotation-based multimodal framework for monitoring both driver and passengers in a car that outperforms state-of-the-art benchmarks in accuracy, precision, and recall while exhibiting superior robustness under limited labeled data and severe missing modality.

Abstract

Recently, in-car monitoring has emerged as a promising technology for detecting early-stage abnormal status of the driver and providing timely alerts to prevent traffic accidents. Although training models with multimodal data enhances the reliability of abnormal status detection, the scarcity of labeled data and the imbalance of class distribution impede the extraction of critical abnormal state features, significantly deteriorating training performance. Furthermore, missing modalities due to environment and hardware limitations further exacerbate the challenge of abnormal status identification. More importantly, monitoring abnormal health conditions of passengers, particularly in elderly care, is of paramount importance but remains underexplored. To address these challenges, we introduce our IC3M, an efficient camera-rotation-based multimodal framework for monitoring both driver and passengers in a car. Our IC3M comprises two key modules: an adaptive threshold pseudo-labeling strategy and a missing modality reconstruction. The former customizes pseudo-labeling thresholds for different classes based on the class distribution, generating class-balanced pseudo labels to guide model training effectively, while the latter leverages crossmodality relationships learned from limited labels to accurately recover missing modalities by distribution transferring from available modalities. Extensive experimental results demonstrate that IC3M outperforms state-of-the-art benchmarks in accuracy, precision, and recall while exhibiting superior robustness under limited labeled data and severe missing modality.

Figures (17)

• Figure 1: The typical scenario of driver monitoring with multiple sensing modalities.
• Figure 2: Performance comparison with different proportions of labelled samples in the total training samples.
• Figure 3: Performance comparison with balanced and imbalanced training dataset with 90% data from normal class and 10% data from abnormal status.
• Figure 4: Performance comparison with heartbeat data missing in different proportions from the overall training dataset.
• Figure 5: Performance comparison with single camera rotation and other solutions for multi-object monitoring where 1,2,3, and 4 represent the driver, the passenger in assistant driver seat, and two passengers in back row.