Transformer-based Fusion of 2D-pose and Spatio-temporal Embeddings for Distracted Driver Action Recognition

TL;DR

A transformer-based fusion architecture to effectively combine 2D-pose features and spatiotemporal features is designed, which performs well on the A2 test set of the 2023 NVIDIA AI City Challenge for naturalistic driving action recognition and achieves the overlap score of the organizer-defined distracted driver behaviour metric.

Abstract

Classification and localization of driving actions over time is important for advanced driver-assistance systems and naturalistic driving studies. Temporal localization is challenging because it requires robustness, reliability, and accuracy. In this study, we aim to improve the temporal localization and classification accuracy performance by adapting video action recognition and 2D human-pose estimation networks to one model. Therefore, we design a transformer-based fusion architecture to effectively combine 2D-pose features and spatio-temporal features. The model uses 2D-pose features as the positional embedding of the transformer architecture and spatio-temporal features as the main input to the encoder of the transformer. The proposed solution is generic and independent of the camera numbers and positions, giving frame-based class probabilities as output. Finally, the post-processing step combines information from different camera views to obtain final predictions and eliminate false positives. The model performs well on the A2 test set of the 2023 NVIDIA AI City Challenge for naturalistic driving action recognition, achieving the overlap score of the organizer-defined distracted driver behaviour metric of 0.5079.

Figures (7)

• Figure 1: Overview of the proposed transformer-based method. From left top to bottom: example images of the "drinking", "reaching behind", and "pick-up from driver side" action classes.
• Figure 2: Overview of the proposed architecture. Left: different camera-view inputs to the pre-processing step. Middle: extracted 2D-pose and spatio-temporal embeddings are supplied to the transformer architecture. 2D-Pose embedding is considered as the "POSEition" embedding of the encoder, while the spatio-temporal embedding is the main input of the encoder. Right: 1D-class probabilities obtained after the MLP head per camera view are analyzed for finding the significant peaks for each class in a video. Note that the modules shown with a bold dashed border are used for training only.
• Figure 3: Top-Down pose estimation.
• Figure 4: Relative distances between hand and face points (left), and set of facial feature points (right).
• Figure 5: Head pose of a driver during a drinking action.