Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation

TL;DR

This paper addresses the realism gap in adversarial robustness evaluation for trajectory prediction by allowing perturbations to both past and future states of the target agent, ensuring dynamic feasibility via a differentiable dynamic model. It introduces a novel false negative collision attack and new realism-based metrics, demonstrating that perturbations limited to past positions yield unrealistic trajectories, while future-state-constrained, control-action perturbations produce pragmatic yet effective attacks that reveal vulnerabilities in state-of-the-art predictors like Trajectron++. The results highlight a trade-off between attack efficiency and realism, and show that a comprehensive robustness evaluation must consider dynamic feasibility and advanced objectives such as FNC. The work motivates broader adoption of realistic adversarial testing to improve reliability and safety of autonomous-vehicle trajectory prediction systems in real-world settings.

Abstract

Trajectory prediction is a key element of autonomous vehicle systems, enabling them to anticipate and react to the movements of other road users. Evaluating the robustness of prediction models against adversarial attacks is essential to ensure their reliability in real-world traffic. However, current approaches tend to focus on perturbing the past positions of surrounding agents, which can generate unrealistic scenarios and overlook critical vulnerabilities. This limitation may result in overly optimistic assessments of model performance in real-world conditions. In this work, we demonstrate that perturbing not just past but also future states of adversarial agents can uncover previously undetected weaknesses and thereby provide a more rigorous evaluation of model robustness. Our novel approach incorporates dynamic constraints and preserves tactical behaviors, enabling more effective and realistic adversarial attacks. We introduce new performance measures to assess the realism and impact of these adversarial trajectories. Testing our method on a state-of-the-art prediction model revealed significant increases in prediction errors and collision rates under adversarial conditions. Qualitative analysis further showed that our attacks can expose critical weaknesses, such as the inability of the model to detect potential collisions in what appear to be safe predictions. These results underscore the need for more comprehensive adversarial testing to better evaluate and improve the reliability of trajectory prediction models for autonomous vehicles.

Figures (3)

• Figure 1: Scenarios of adversarial attacks by the target agent on trajectory prediction model of the ego agent (AV). A) In the baseline, non-adversarial scenario, based on the observed past trajectory $X = \{X_{\text{ego}}, X_{\text{tar}}\}$ of the target agent, the ego agent makes a prediction $\widehat{Y}$ that aims to closely capture the (unknown) future states $Y$. B) Adversarial (target) agent aims to attack the trajectory prediction model of the AV with the goal for that model to produce an erroneous prediction of the target agent. Existing approaches in the literature typically generate adversarial attacks by perturbing past states of the target vehicle $\widetilde{X}$ in such a way that the corresponding prediction $\widehat{\widetilde{Y}}$ deviates as much as possible from the ground-truth future states $Y$. However, existing methods do not guarantee that these future states $Y$ actually lie in the reachable set of the target agent after the perturbation, which can result in an overall unsuccessful attack where $\widehat{\widetilde{Y}}$ is actually consistent with the actual future of the adversarial agent. C) In our proposed approach, we generate (and constrain) perturbations not only for the past states $\widetilde{X}$ but also future states $\widetilde{Y}$ of the target agent. This ensures that the observed future $Y$ stays dynamically feasible, which ensures that prediction errors resulting from the perturbation are meaningful. D) Our approach allows us to generate a novel type of adversarial attacks: false negative collision attack, where the adversarial agent takes a future trajectory that collides with the AV but deceives the AV into predicting a non-colliding trajectory. In B) - D), the yellow area represent the dynamically reachable set starting from the first observation, while the blue region represents the same starting from the prediction point. The green region represents a simple distance constraint often employed in adversarial attacks in the literature.
• Figure 2: Generating adversarial trajectories with constraining future states using control actions. 1) During initialization, we extract the control actions ($U_{\text{tar}} = \widetilde{U}^0_{\text{tar}}$, $V_{\text{tar}} = \widetilde{V}_{\text{tar}}^0$) from the unperturbed trajectory parts $X_{\text{tar}}$ and $Y_{\text{tar}}$ respectively. 2) Afterwards, we iteratively perturb those control actions, where, starting with the given perturbed control actions $\widetilde{U}_{\text{tar}}^m$ and $\widetilde{V}_{\text{tar}}^m$, we get the corresponding perturbed trajectories $\widetilde{X}_{\text{tar}}^m$ and $\widetilde{Y}_{\text{tar}}^m$. 3) Based thereon -- together with the prediction $\widehat{\widetilde{Y}}^m_{\text{tar}}$ from updated states $\widetilde{X}_{\text{tar}}^m$ -- we can calculate the adversarial loss $L_m$ and its derivative, with which the perturbation can be updated. The constraints $C$ in this update are represented by the purple dotted line for the absolute control limits and the blue dashed line for the relative bounds. 4) This procedure is repeated for $M_{\max}$ iterations. 5) In the end, the final perturbed trajectories are extracted.
• Figure 3: Three regularization approaches for preserving tactical behavior. The black line represents the ground truth trajectory, with the colored arrows showing the direction in which the regularization penalties are applied. In A), the different colors represent that the different penalties are applied to different timesteps.