Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting

TL;DR

This work tackles 3D pedestrian pose forecasting in autonomous driving by conditioning predictions on surrounding vehicles. It extends the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes and a scene-aware sampling scheme, and introduces a Vehicle-conditioned 3D pose forecasting network that augments the TBIFormer backbone with a Vehicle Encoder and Pedestrian–Vehicle Interaction Cross-Attention. The approach yields consistent improvements across one-, two-, and three-pedestrian scenarios, with larger gains as interaction complexity grows, and ablation studies confirm the value of explicit vehicle modeling and TRPE-enhanced cross-attention. Overall, incorporating vehicle context enhances pose forecasting accuracy, contributing to safer planning in multi-agent urban environments. $MPJPE$, $APE$, and $FDE$ improvements demonstrate practical impact for real-world autonomous driving systems.

Abstract

Accurately predicting pedestrian motion is crucial for safe and reliable autonomous driving in complex urban environments. In this work, we present a 3D vehicle-conditioned pedestrian pose forecasting framework that explicitly incorporates surrounding vehicle information. To support this, we enhance the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes, enabling realistic modeling of multi-agent pedestrian-vehicle interactions. We introduce a sampling scheme to categorize scenes by pedestrian and vehicle count, facilitating training across varying interaction complexities. Our proposed network adapts the TBIFormer architecture with a dedicated vehicle encoder and pedestrian-vehicle interaction cross-attention module to fuse pedestrian and vehicle features, allowing predictions to be conditioned on both historical pedestrian motion and surrounding vehicles. Extensive experiments demonstrate substantial improvements in forecasting accuracy and validate different approaches for modeling pedestrian-vehicle interactions, highlighting the importance of vehicle-aware 3D pose prediction for autonomous driving. Code is available at: https://github.com/GuangxunZhu/VehCondPose3D

Figures (5)

• Figure 1: Illustration of our Vehicle-conditioned pedestrian pose forecasting. Pedestrian predictions are not only based on their historical motion but are also influenced by surrounding vehicle information.
• Figure 2: Illustration of pedestrians and vehicles in a scene with their trajectories and inter-agent distances.
• Figure 3: Number of pedestrian–vehicle interaction training samples under varying numbers of surrounding vehicles and different distance thresholds for scenarios ranging from one to three pedestrians.
• Figure 4: Overview of the proposed Vehicle-conditioned pedestrian pose forecasting network. The model receives 3D pedestrian poses and 3D vehicle bounding box data, transforms them into displacement sequences, and applies Discrete Cosine Transform (DCT) to discard high-frequency components for a more compact representation. Pedestrian and vehicle features are then processed through separate encoders and fused via cross-attention before being decoded into future pedestrian poses.
• Figure 5: Qualitative comparison between TBIFormer peng2023trajectory and our model. Ground truth trajectories are shown in green, TBIFormer predictions in blue, and ours in red. Black boxes denote surrounding vehicles used by our model as input, whereas TBIFormer does not use vehicle information.