Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatial-Temporal Perception with Causal Inference for Naturalistic Driving Action Recognition

Qing Chang, Wei Dai, Zhihao Shuai, Limin Yu, Yutao Yue

TL;DR

This work tackles naturalistic driving action recognition by introducing Spatial-Temporal Perception (STP), a single-modality RGB framework that jointly models temporal dynamics and spatial relationships between key points. It combines dual-feature extraction, context joint encoding via a stacked multi-scale channel transformer, and a causal-aware decoder to perform behavior recognition and temporal action localization, optimized by likelihood over factorization orders. The approach achieves state-of-the-art results on Drive&Act and SynDD2 while maintaining efficiency and eliminating the need for multimodal inputs. The proposed method holds practical potential for real-time vehicle cabin monitoring and safer driver-vehicle interactions by accurately identifying subtle actions and their temporal boundaries.

Abstract

Naturalistic driving action recognition is essential for vehicle cabin monitoring systems. However, the complexity of real-world backgrounds presents significant challenges for this task, and previous approaches have struggled with practical implementation due to their limited ability to observe subtle behavioral differences and effectively learn inter-frame features from video. In this paper, we propose a novel Spatial-Temporal Perception (STP) architecture that emphasizes both temporal information and spatial relationships between key objects, incorporating a causal decoder to perform behavior recognition and temporal action localization. Without requiring multimodal input, STP directly extracts temporal and spatial distance features from RGB video clips. Subsequently, these dual features are jointly encoded by maximizing the expected likelihood across all possible permutations of the factorization order. By integrating temporal and spatial features at different scales, STP can perceive subtle behavioral changes in challenging scenarios. Additionally, we introduce a causal-aware module to explore relationships between video frame features, significantly enhancing detection efficiency and performance. We validate the effectiveness of our approach using two publicly available driver distraction detection benchmarks. The results demonstrate that our framework achieves state-of-the-art performance.

Spatial-Temporal Perception with Causal Inference for Naturalistic Driving Action Recognition

TL;DR

This work tackles naturalistic driving action recognition by introducing Spatial-Temporal Perception (STP), a single-modality RGB framework that jointly models temporal dynamics and spatial relationships between key points. It combines dual-feature extraction, context joint encoding via a stacked multi-scale channel transformer, and a causal-aware decoder to perform behavior recognition and temporal action localization, optimized by likelihood over factorization orders. The approach achieves state-of-the-art results on Drive&Act and SynDD2 while maintaining efficiency and eliminating the need for multimodal inputs. The proposed method holds practical potential for real-time vehicle cabin monitoring and safer driver-vehicle interactions by accurately identifying subtle actions and their temporal boundaries.

Abstract

Naturalistic driving action recognition is essential for vehicle cabin monitoring systems. However, the complexity of real-world backgrounds presents significant challenges for this task, and previous approaches have struggled with practical implementation due to their limited ability to observe subtle behavioral differences and effectively learn inter-frame features from video. In this paper, we propose a novel Spatial-Temporal Perception (STP) architecture that emphasizes both temporal information and spatial relationships between key objects, incorporating a causal decoder to perform behavior recognition and temporal action localization. Without requiring multimodal input, STP directly extracts temporal and spatial distance features from RGB video clips. Subsequently, these dual features are jointly encoded by maximizing the expected likelihood across all possible permutations of the factorization order. By integrating temporal and spatial features at different scales, STP can perceive subtle behavioral changes in challenging scenarios. Additionally, we introduce a causal-aware module to explore relationships between video frame features, significantly enhancing detection efficiency and performance. We validate the effectiveness of our approach using two publicly available driver distraction detection benchmarks. The results demonstrate that our framework achieves state-of-the-art performance.

Paper Structure

This paper contains 11 sections, 6 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Context Joint Encoding
  5. Causal Query Decoding
  6. Experiments and Results
  7. Datasets and Evaluation Metrics
  8. Implementation Details
  9. Quantitative Results
  10. Ablation Study and Visualization
  11. Conclusion

Figures (3)

  • Figure 1: Illustrations of challenging cases in driving action recognition. (a) Calling and (b) drinking scenes, where the objects are partially visible and the lighting conditions are unstable. (c) Variations in the distance $d$ between key points assist in identifying behavior categories and temporal localization.
  • Figure 2: The overall architecture of STP.
  • Figure 3: Causal-aware module incrementally exposes video frames to learnable queries to decouple spatial and temporal features.