Generalizability Analysis of Graph-based Trajectory Predictor with Vectorized Representation

TL;DR

An in-depth generalizability analysis of one of the state-of-the-art graph-based trajectory predictors that utilize vectorized representation shows significant performance degradation due to domain shift, and feature attribution provides insights to identify potential causes of these problems.

Abstract

Trajectory prediction is one of the essential tasks for autonomous vehicles. Recent progress in machine learning gave birth to a series of advanced trajectory prediction algorithms. Lately, the effectiveness of using graph neural networks (GNNs) with vectorized representations for trajectory prediction has been demonstrated by many researchers. Nonetheless, these algorithms either pay little attention to models' generalizability across various scenarios or simply assume training and test data follow similar statistics. In fact, when test scenarios are unseen or Out-of-Distribution (OOD), the resulting train-test domain shift usually leads to significant degradation in prediction performance, which will impact downstream modules and eventually lead to severe accidents. Therefore, it is of great importance to thoroughly investigate the prediction models in terms of their generalizability, which can not only help identify their weaknesses but also provide insights on how to improve these models. This paper proposes a generalizability analysis framework using feature attribution methods to help interpret black-box models. For the case study, we provide an in-depth generalizability analysis of one of the state-of-the-art graph-based trajectory predictors that utilize vectorized representation. Results show significant performance degradation due to domain shift, and feature attribution provides insights to identify potential causes of these problems. Finally, we conclude the common prediction challenges and how weighting biases induced by the training process can deteriorate the accuracy.

Figures (5)

• Figure 1: Illustration of the analysis framework. Objects in a traffic scenario are represented as a set of vectors through segmentation and encoding. The forward trajectory prediction takes these vectors as inputs to the model and derives results. The backward feature attribution estimates how the model attributes its prediction accuracy to the input vectors based on the predictions.
• Figure 2: Example visualization of performance differences between the actual input and generated baselines. The proposed baseline (blue curve) generated using different standard deviations has significantly lower accuracy regarding the actual input than the others.
• Figure 3: Selected best-case (top row) and typical worst-case prediction (bottom row) results. Input vectors (i.e. map polylines and history trajectories) are colored by their absolute values of the sum integrated gradients over the feature dimension. A darker coloring indicates a higher relevance to the final prediction. Training and testing settings are: (a) model trained on CHN Merging ZS0 and tested on the same scenario; (b) model trained on CHN Merging ZS0 and tested on USA Roundabout FT; (c) model trained on USA Intersection MA and tested on DEU Roundabout OF; (d) model trained on CHN Merging ZS2 and tested on USA Intersection MA; (e) model trained on DEU Roundabout OF and tested on the same scenario; (f) model trained on USA Intersection GL and tested on USA Intersection MA.
• Figure 4: Average feature attribution ratios aggregated by polyline types with respect to different test cases in USA Intersection GL and total feature attribution (right) of models trained on CHN Merging ZS2 and USA Roundabout FT.
• Figure 5: Average feature attribution ratios aggregated by node features types with respect to different test cases in USA Intersection GL and total feature attribution (right) of models trained on CHN Merging ZS2 and USA Roundabout FT.