Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness

TL;DR

This paper investigates Out-of-Distribution (OoD) robustness in autonomous-vehicle trajectory prediction by cross-dataset evaluation between Argoverse 2 (A2) and Waymo Open Motion (WO). It compares three state-of-the-art models with matched In-Distribution (ID) performance across different inductive biases, data representations, and augmentation strategies, in both A2→WO and WO→A2 settings, after homogenizing the datasets. The authors find that a small, highly biased polynomial model (EP-Q) yields the strongest OoD generalization when trained on the smaller A2, but all models show poor OoD performance when trained on the larger WO, revealing limits of data quantity alone for robustness. They propose two potential drivers—task complexity and dataset noise—and argue for OoD testing as a core evaluation criterion alongside traditional ID metrics. The results guide design choices for trajectory prediction models and benchmarks, highlighting that architectural bias and data properties jointly shape cross-domain generalization.

Abstract

We study the Out-of-Distribution (OoD) generalization ability of three SotA trajectory prediction models with comparable In-Distribution (ID) performance but different model designs. We investigate the influence of inductive bias, size of training data and data augmentation strategy by training the models on Argoverse 2 (A2) and testing on Waymo Open Motion (WO) and vice versa. We find that the smallest model with highest inductive bias exhibits the best OoD generalization across different augmentation strategies when trained on the smaller A2 dataset and tested on the large WO dataset. In the converse setting, training all models on the larger WO dataset and testing on the smaller A2 dataset, we find that all models generalize poorly, even though the model with the highest inductive bias still exhibits the best generalization ability. We discuss possible reasons for this surprising finding and draw conclusions about the design and test of trajectory prediction models and benchmarks.

Figures (3)

• Figure 1: The two augmentation strategies for non-focal agent data employed in benchmark models. Left: FMAE employs heterogeneous augmentation, representing information in the focal agent's coordinate frame and only making uni-modal predictions for non-focal agents. Right: QCNet employs homogeneous augmentation, encoding information in each agent's individual coordinate frame and making multi-modal predictions for both focal and non-focal agents alike.
• Figure 2: The OoD testing results of FMAE, QCNet, EP and their variants. We indicate the absolute and relative difference in displacement error between ID and OoD results. The solid bars represent ID results reported in Table \ref{['tab: in-distribution result combined']}, while the transparent bars indicate the increase in displacement error during OoD testing. Left: Models trained on the homogenized WO training set and tested on the homogenized WO (ID) and A2 (OoD) validation sets. Right: Models trained on the homogenized A2 training set and tested on the homogenized A2 (ID) and WO (OoD) validation sets yao_improving_2024.
• Figure 3: Kernel density plot of normalized longitudinal and lateral distances with 1.1-second and 5-second history lengths for Waymo Motion validation set. Contours indicate the 30-th, 60-th and 90-th percentiles, respectively, illustrating a broader spread for the prediction task with a 1.1-second history.