Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transfer-LMR: Heavy-Tail Driving Behavior Recognition in Diverse Traffic Scenarios

Chirag Parikh, Ravi Shankar Mishra, Rohan Chandra, Ravi Kiran Sarvadevabhatla

TL;DR

Transfer-LMR tackles heavy-tail driving behavior recognition in low-resource dashcam datasets by a two-stage training pipeline that first learns general video features and then refines tail-class representations with Long-Tailed Mixed Reconstruction (LMR). It introduces boundary-constrained data augmentation to preserve peripheral context, temporal feature aggregation, and LMR-based feature refinement to diversify tail-class features without sacrificing overall mean average precision. Evaluated on METEOR and HDD, it delivers substantial gains in average class accuracy across multiple backbones, notably improving tail-class predictions such as CT, WL, and DT, and achieves state-of-the-art results with I3D pre-trained on Kinetics-400. The method demonstrates backbone-agnostic applicability and practical inference speed, making it a robust solution for robust planning and safety in diverse traffic scenarios.

Abstract

Recognizing driving behaviors is important for downstream tasks such as reasoning, planning, and navigation. Existing video recognition approaches work well for common behaviors (e.g. "drive straight", "brake", "turn left/right"). However, the performance is sub-par for underrepresented/rare behaviors typically found in tail of the behavior class distribution. To address this shortcoming, we propose Transfer-LMR, a modular training routine for improving the recognition performance across all driving behavior classes. We extensively evaluate our approach on METEOR and HDD datasets that contain rich yet heavy-tailed distribution of driving behaviors and span diverse traffic scenarios. The experimental results demonstrate the efficacy of our approach, especially for recognizing underrepresented/rare driving behaviors.

Transfer-LMR: Heavy-Tail Driving Behavior Recognition in Diverse Traffic Scenarios

TL;DR

Transfer-LMR tackles heavy-tail driving behavior recognition in low-resource dashcam datasets by a two-stage training pipeline that first learns general video features and then refines tail-class representations with Long-Tailed Mixed Reconstruction (LMR). It introduces boundary-constrained data augmentation to preserve peripheral context, temporal feature aggregation, and LMR-based feature refinement to diversify tail-class features without sacrificing overall mean average precision. Evaluated on METEOR and HDD, it delivers substantial gains in average class accuracy across multiple backbones, notably improving tail-class predictions such as CT, WL, and DT, and achieves state-of-the-art results with I3D pre-trained on Kinetics-400. The method demonstrates backbone-agnostic applicability and practical inference speed, making it a robust solution for robust planning and safety in diverse traffic scenarios.

Abstract

Recognizing driving behaviors is important for downstream tasks such as reasoning, planning, and navigation. Existing video recognition approaches work well for common behaviors (e.g. "drive straight", "brake", "turn left/right"). However, the performance is sub-par for underrepresented/rare behaviors typically found in tail of the behavior class distribution. To address this shortcoming, we propose Transfer-LMR, a modular training routine for improving the recognition performance across all driving behavior classes. We extensively evaluate our approach on METEOR and HDD datasets that contain rich yet heavy-tailed distribution of driving behaviors and span diverse traffic scenarios. The experimental results demonstrate the efficacy of our approach, especially for recognizing underrepresented/rare driving behaviors.
Paper Structure (18 sections, 9 equations, 5 figures, 2 tables)

This paper contains 18 sections, 9 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Work
  4. Ego Vehicle Driving Behavior Recognition
  5. Long-Tailed Recognition
  6. Background
  7. Problem Formulation
  8. Long-Tailed Mixed Reconstruction (LMR)
  9. Transfer-LMR
  10. Data Augmentation
  11. Two-Stage Training Routine
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Baselines
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Heay-Tail Distributed ego-vehicle driving behavior recognition in diverse traffic environments. Our proposed approach, Transfer-LMR, significantly outperforms existing behavior recognition methods by 9.2% and 2.9% in average class accuracy on real-world driving datasets METEORchandra2023meteor and HDDramanishka2018toward.
  • Figure 2: TransferLMR: A Two-Stage Training Method for Heavy-Tail Driving Behavior Recognition. Instance-balanced (IB) and class-balanced (CB) are sampling strategies. DA is our customized data augmentation method. Averaging (Avg.) is used for Temporal Feature Aggregation. Long-tailed Mixed Reconstruction (LMR) is used for Feature Refinement. Driving Behavior Class prediction in the Testing Stage does not require LMR.
  • Figure 3: Modified random resized cropping augmentation method tailored for dashcam-based driving videos. The dotted orange line represents the position and relative dimensions of the minimum cropping region obtained from this method, while the dotted blue line corresponds to that obtained from the original unaltered augmentation method.
  • Figure 4: Qualitative Results for Driving Behavior Recognition on METEOR dataset chandra2023meteor. The key frame of sample driving videos from the validation set of METEOR dataset is shown. Their Bird's Eye View (BEV) is also displayed demonstrating the vehicle trajectories that define the ego's driving behavior when observed in the video's full duration. The behavior predictions of CE and cRT baselines are compared with Transfer-LMR (TLMR) method when using the (Kinetics-400 pre-trained) I3D video backbone.
  • Figure 5: Qualitative Results for Driving Behavior Recognition on HDD dataset ramanishka2018toward. The key frame of sample driving videos from the validation set of HDD dataset is shown. Their Bird's Eye View (BEV) is also displayed demonstrating the vehicle trajectories that define the ego's driving behavior when observed in the video's full duration. The behavior predictions of the CE baseline are compared with the Transfer-LMR (TLMR) method when using the (Kinetics-400 pre-trained) I3D video backbone.