Super Agents and Confounders: Influence of surrounding agents on vehicle trajectory prediction

Abstract

In highly interactive driving scenes, trajectory prediction is conditioned on information from surrounding traffic participants such as cars and pedestrians. Our main contribution is a comprehensive analysis of state-of-the-art trajectory predictors, which reveals a surprising and critical flaw: many surrounding agents degrade prediction accuracy rather than improve it. Using Shapley-based attribution, we rigorously demonstrate that models learn unstable and non-causal decision-making schemes that vary significantly across training runs. Building on these insights, we propose to integrate a Conditional Information Bottleneck (CIB), which does not require additional supervision and is trained to effectively compress agent features as well as ignore those that are not beneficial for the prediction task. Comprehensive experiments using multiple datasets and model architectures demonstrate that this simple yet effective approach not only improves overall trajectory prediction performance in many cases but also increases robustness to different perturbations. Our results highlight the importance of selectively integrating contextual information, which can often contain spurious or misleading signals, in trajectory prediction. Moreover, we provide interpretable metrics for identifying non-robust behavior and present a promising avenue towards a solution.

Figures (5)

• Figure 1: (a) Removing a "Confounding Agent" can significantly improve prediction accuracy. (b) The Insertion Test based on the attribution analysis reveals a U-shaped performance curve, where using only the subset of the most helpful agents yields the best results.
• Figure 2: Overview of the proposed prediction architecture. The Conditional Information Bottleneck (CIB) module compresses surrounding agent features while conditioning on the target agent's state to extract relevant information. These are then fused with lane geometry and target agent embeddings within the Interactor before final trajectory decoding.
• Figure 3: Insertion test performed on the train and validation set of nuScenes. On the validation dataset, the confounding agents have a much stronger effect compared to the training set, almost canceling out the Super Agents.
• Figure 4: Consistency of improving agent influence for the QEANet model on nuScenes. (a) Intra-model Agreement: A single trained model shows high consistency in agent attributions across five inference seeds. (b) Inter-model Disagreement: Models trained with different random seeds show significant variability, with agent influence profiles following a random baseline, indicating unstable learned decision-making.
• Figure 5: Consistency of improving agent influence for the MTR model and the MTR+IB model compared to human labels. (a) Causal Agents: Labeled Causal Agents are only slightly more likely to have a performance-improving influence. (b) Non-Causal Agents: The influence of Non-Causal Agents follows the random baseline, highlighting the models inability to identify them. The same analysis is done for the MTR+IB Model: (c) There is no increase in the number of Causal Agents used as improving agents but the average attribution value of agents with $r_{\text{NLL}}=\frac{5}{5}$ decreased significantly (d) The usage of Non-Causal Agents is still in line with the random baseline.