Automated Vehicle Driver Monitoring Dataset from Real-World Scenarios

TL;DR

The paper introduces the JKU-ITS Automated Vehicle Driver Monitoring (AVDM) dataset, the first real-world driver activity collection for SAE Level 3+ automated driving, captured from a moving test vehicle under varied illumination and weather. It provides 17 participants performing 8 actions across roughly 200 minutes of RGB video, with dataset formats supporting 6-class and 8-class labels and a semi-automatic labeling tool. A baseline Inflated 3D (I3D) network is trained on 64-frame snippets to benchmark action recognition, achieving high accuracy on several classes but facing confusions due to occlusion and proximity of hands to the steering wheel. The work establishes AVDM as a realistic benchmark for driver monitoring in automated driving and outlines future expansions to improve generalization and safety applications. The dataset is openly accessible on IEEE Dataport, facilitating community-driven advancement in real-world driver activity recognition with varied environmental conditions.

Abstract

From SAE Level 3 of automation onwards, drivers are allowed to engage in activities that are not directly related to driving during their travel. However, in level 3, a misunderstanding of the capabilities of the system might lead drivers to engage in secondary tasks, which could impair their ability to react to challenging traffic situations. Anticipating driver activity allows for early detection of risky behaviors, to prevent accidents. To be able to predict the driver activity, a Deep Learning network needs to be trained on a dataset. However, the use of datasets based on simulation for training and the migration to real-world data for prediction has proven to be suboptimal. Hence, this paper presents a real-world driver activity dataset, openly accessible on IEEE Dataport, which encompasses various activities that occur in autonomous driving scenarios under various illumination and weather conditions. Results from the training process showed that the dataset provides an excellent benchmark for implementing models for driver activity recognition.

Figures (6)

• Figure 1: An image from the JKU-ITS AVDM dataset, showcasing a realistic driver reaction. Despite being tasked with reading a newspaper, the driver periodically monitors their surroundings to avoid hazardous situations.
• Figure 2: JKU-ITS research vehicle employed for the data collection.
• Figure 3: Image examples from the JKU-ITS AVDM dataset showcasing a variety of activities captured under varying illumination conditions.
• Figure 4: Exemplary images from the AVDM dataset for predicting actions using the I3D model with the 6 labelled classes version of the AVDM dataset. Labels in the bottom left corner of each image are color-coded as: white for ground truth, green for correct classifications, and red for incorrect classifications. In (a), we showcase instances from the test set where the network accurately predicted the actions; (b) illustrates a challenging scenario where distractions in the environment pose difficulties for the DL network: the proximity of the hands to the steering wheel might mislead the network into predicting driving instead of sitting still; (c) presents a test sample depicting the similarity between two closely related classes: reading a book and reading a magazine. Due to the resemblance in posture and the items being used, the network finds it easier to detect the general activity rather than classifying the specific type of object being read.
• Figure 5: The labeling tool developed accepts the video path as input, allowing users to navigate through individual frames of the video. Users can easily switch between assigned labels for each frame and utilize the tool to automatically label frames with the selected label.