DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset

TL;DR

This work tackles fine-grained driver action recognition by integrating object-level cues and human–object relations within car cabins. It introduces the DAOS dataset, a large, multi-view, multi-modal resource with dense object annotations aligned to actions, and proposes AOR-Net, a Chain-of-action prompting model with a Mixture of Thoughts that fuses action, object, and relation information via textual prototypes. The approach yields state-of-the-art results on DAOS, demonstrating the value of explicit relation reasoning for robust action understanding under varying object contexts and lighting. The dataset and model together offer a practical path toward more reliable, real-time driver monitoring that leverages object semantics to disambiguate subtle actions.

Abstract

In driver activity monitoring, movements are mostly limited to the upper body, which makes many actions look similar. To tell these actions apart, human often rely on the objects the driver is using, such as holding a phone compared with gripping the steering wheel. However, most existing driver-monitoring datasets lack accurate object-location annotations or do not link objects to their associated actions, leaving a critical gap for reliable action recognition. To address this, we introduce the Driver Action with Object Synergy (DAOS) dataset, comprising 9,787 video clips annotated with 36 fine-grained driver actions and 15 object classes, totaling more than 2.5 million corresponding object instances. DAOS offers multi-modal, multi-view data (RGB, IR, and depth) from front, face, left, and right perspectives. Although DAOS captures a wide range of cabin objects, only a few are directly relevant to each action for prediction, so focusing on task-specific human-object relations is essential. To tackle this challenge, we propose the Action-Object-Relation Network (AOR-Net). AOR-Net comprehends complex driver actions through multi-level reasoning and a chain-of-action prompting mechanism that models the logical relationships among actions, objects, and their relations. Additionally, the Mixture of Thoughts module is introduced to dynamically select essential knowledge at each stage, enhancing robustness in object-rich and object-scarce conditions. Extensive experiments demonstrate that our model outperforms other state-of-the-art methods on various datasets.

Figures (8)

• Figure 1: Comparison of the existing Driver Monitoring dataset and our DAOS. Our dataset includes precise object annotations, multimodal calibration, various lighting conditions and multi-view videos on all three modalities.
• Figure 2: Comparison of model architectures for driver action recognition. (Top left) Traditional action recognition models rely solely on visual encoders to classify actions from raw video frames, lacking object awareness. (Top right) Object-augmented action recognition models incorporate additional object information (e.g., bounding boxes or cropped regions) to enrich visual features via feature fusion. (Bottom) The proposed Action–Object–Relation Network (AOR-Net) further extends this paradigm by explicitly modeling human–object relations through relation reasoning and multilevel fusion Mixture-of-Thoughts (MoT), enabling a deeper understanding of driver intent in complex cabin environments.
• Figure 3: Statistics of the proposed DAOS dataset. (a) Hierarchical action labels: at the coarse level, consecutive actions involving the same object are merged into 12 major categories; at the fine‑grained level, those are split into 36 more detailed action classes. (b) Action label data distribution on a logarithmic scale. (c) Object instance data distribution on a logarithmic scale.
• Figure 4: Visualization and setup of the proposed DAOS dataset. (a) Car cabin setup with four Azure Kinect devices (front, facial, left, right) capturing RGB, Depth/IR under LED and NIR illumination. (b) Multimodal multiview image samples from the four camera views (Left, Facial, Front, Right) and three sensing modalities (RGB, IR, Depth).
• Figure 5: Left: Data distribution comparison of lighting conditions for existing and our proposed dataset. Right: Image samples for different lighting conditions, covering sunny, rainy, cloudy and night conditions. Lighting condition settings ensures comprehensive coverage of in-cabin scenarios.