Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VIT-Ped: Visionary Intention Transformer for Pedestrian Behavior Analysis

Aly R. Elkammar, Karim M. Gamaleldin, Catherine M. Elias

TL;DR

This work tackles pedestrian crossing intention prediction to improve safety for autonomous driving as systems progress from level 3 to level 4. It introduces VIT-Ped, a multi-modal transformer framework that blends a Video Vision Transformer (ViViT) for visual streams (local surround, local context, and global context) with a vanilla Transformer for non-visual inputs (bounding boxes, centers, pose keypoints, and ego-vehicle speed), combined through multiple fusion strategies. Evaluations on the JAAD dataset demonstrate state-of-the-art performance across key metrics (Accuracy, AUC, F1), with ablations highlighting the contributions of each modality and fusion design, and both a compact ViViT-based model and a non-visual-only model achieving SOTA with substantial size reductions. The results suggest practical impact for real-time driving systems, offering robust cross-scenario pedestrian intent prediction and supporting safer ego-vehicle decision-making in varied environments.

Abstract

Pedestrian Intention prediction is one of the key technologies in the transition from level 3 to level 4 autonomous driving. To understand pedestrian crossing behaviour, several elements and features should be taken into consideration to make the roads of tomorrow safer for everybody. We introduce a transformer / video vision transformer based algorithm of different sizes which uses different data modalities .We evaluated our algorithms on popular pedestrian behaviour dataset, JAAD, and have reached SOTA performance and passed the SOTA in metrics like Accuracy, AUC and F1-score. The advantages brought by different model design choices are investigated via extensive ablation studies.

VIT-Ped: Visionary Intention Transformer for Pedestrian Behavior Analysis

TL;DR

This work tackles pedestrian crossing intention prediction to improve safety for autonomous driving as systems progress from level 3 to level 4. It introduces VIT-Ped, a multi-modal transformer framework that blends a Video Vision Transformer (ViViT) for visual streams (local surround, local context, and global context) with a vanilla Transformer for non-visual inputs (bounding boxes, centers, pose keypoints, and ego-vehicle speed), combined through multiple fusion strategies. Evaluations on the JAAD dataset demonstrate state-of-the-art performance across key metrics (Accuracy, AUC, F1), with ablations highlighting the contributions of each modality and fusion design, and both a compact ViViT-based model and a non-visual-only model achieving SOTA with substantial size reductions. The results suggest practical impact for real-time driving systems, offering robust cross-scenario pedestrian intent prediction and supporting safer ego-vehicle decision-making in varied environments.

Abstract

Pedestrian Intention prediction is one of the key technologies in the transition from level 3 to level 4 autonomous driving. To understand pedestrian crossing behaviour, several elements and features should be taken into consideration to make the roads of tomorrow safer for everybody. We introduce a transformer / video vision transformer based algorithm of different sizes which uses different data modalities .We evaluated our algorithms on popular pedestrian behaviour dataset, JAAD, and have reached SOTA performance and passed the SOTA in metrics like Accuracy, AUC and F1-score. The advantages brought by different model design choices are investigated via extensive ablation studies.
Paper Structure (1 section, 6 equations, 3 figures, 2 tables)

This paper contains 1 section, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Results

Figures (3)

  • Figure 1: Overview on the VIT-Ped Architecture
  • Figure 2: Our model predicts that the intention of the pedestrian correctly unlike PCPA. Red bounding box means the pedestrian is about to cross and green means not crossing.
  • Figure 3: Chaotic Environment. The frames are separated from one another by a window of 1-2 seconds. Red bounding box means the pedestrian is about to cross/crossing and green means not crossing.