Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback

TL;DR

The paper tackles open-loop trajectory prediction in automated driving, where successive predictions fail to leverage past errors. It introduces retrospection, a closed-loop training framework, with two attention-based modules (Ret-S and Ret-C) that analyze past prediction errors stored in a rolling buffer to adjust future predictions. Evaluations on nuScenes and Argoverse show substantial minADE reductions (up to $31.9\%$) and improved robustness in scenarios with incomplete detections, underscoring the value of error feedback during inference. The work advances trajectory forecasting by enabling models to learn from their mistakes over time, with potential safety gains for real-world autonomous driving systems.

Abstract

In automated driving, predicting trajectories of surrounding vehicles supports reasoning about scene dynamics and enables safe planning for the ego vehicle. However, existing models handle predictions as an instantaneous task of forecasting future trajectories based on observed information. As time proceeds, the next prediction is made independently of the previous one, which means that the model cannot correct its errors during inference and will repeat them. To alleviate this problem and better leverage temporal data, we propose a novel retrospection technique. Through training on closed-loop rollouts the model learns to use aggregated feedback. Given new observations it reflects on previous predictions and analyzes its errors to improve the quality of subsequent predictions. Thus, the model can learn to correct systematic errors during inference. Comprehensive experiments on nuScenes and Argoverse demonstrate a considerable decrease in minimum Average Displacement Error of up to 31.9% compared to the state-of-the-art baseline without retrospection. We further showcase the robustness of our technique by demonstrating a better handling of out-of-distribution scenarios with undetected road-users.

Figures (6)

• Figure 1: Learning through retrospection during inference. Feedback on how good previously made predictions match with the subsequently measured data improves the prediction quality substantially whereas conventional prediction handles consecutive predictions as independent tasks and does not improve over time. The models were trained and evaluated on nuScenes across seven consecutive predictions.
• Figure 2: Advantages of closed-loop training in automated driving. Whereas many methods use closed-loop simulation to extend the distribution of training scenarios, others aggregate latent features across multiple consecutive timesteps. We propose retrospection to account for previous errors in the next prediction.
• Figure 3: Learning through retrospection. (a) Model overview: the prediction model stores its previous predictions and measurements in a buffer. During training, the “measured” trajectory corresponds to the ground truth trajectory up to the current timestep (implemented as a temporal mask). A retrospection module learns to refine the trajectory by analyzing its previous predictions and how well they matched the subsequent measurements to correct systematic errors. This mechanism is then applicable at testtime. Two different retrospection modules are compared: (b) Ret-S, which is based on self-attention of the buffered errors and (c) Ret-C, which is based on cross-attention between the current prediction and the buffer.
• Figure 4: Masked error buffer and its conversion into an error sequence. Each buffer entry $E^r_f$ contains the concatenation of real measurement, prediction, and their pointwise distance. During inference, the masked entries are unavailable since the corresponding information was not measured yet, whereas during training the entries are zero-padded to prevent future information leakage. An MLP maps all entries of one rollout step $r$ into a single token to enable attention-based retrospection of the resulting error sequence.
• Figure 5: Data structure to enable closed-loop rollouts. We extract $R$ consecutive samples from each scenario, comprising agent track histories and ground truth future.