Multi-modal Motion Prediction using Temporal Ensembling with Learning-based Aggregation

TL;DR

Temporal Ensembling with Learning-based Aggregation with Learning-based Aggregation is introduced, a meta-algorithm designed to mitigate the issue of missing behaviors in trajectory prediction, which leads to inconsistent predictions across consecutive frames.

Abstract

Recent years have seen a shift towards learning-based methods for trajectory prediction, with challenges remaining in addressing uncertainty and capturing multi-modal distributions. This paper introduces Temporal Ensembling with Learning-based Aggregation, a meta-algorithm designed to mitigate the issue of missing behaviors in trajectory prediction, which leads to inconsistent predictions across consecutive frames. Unlike conventional model ensembling, temporal ensembling leverages predictions from nearby frames to enhance spatial coverage and prediction diversity. By confirming predictions from multiple frames, temporal ensembling compensates for occasional errors in individual frame predictions. Furthermore, trajectory-level aggregation, often utilized in model ensembling, is insufficient for temporal ensembling due to a lack of consideration of traffic context and its tendency to assign candidate trajectories with incorrect driving behaviors to final predictions. We further emphasize the necessity of learning-based aggregation by utilizing mode queries within a DETR-like architecture for our temporal ensembling, leveraging the characteristics of predictions from nearby frames. Our method, validated on the Argoverse 2 dataset, shows notable improvements: a 4% reduction in minADE, a 5% decrease in minFDE, and a 1.16% reduction in the miss rate compared to the strongest baseline, QCNet, highlighting its efficacy and potential in autonomous driving.

Figures (6)

• Figure 1: Illustration of Missing Behaviors Issue - Missing behaviors refer to cases where predictions are occasionally wrong, resulting in inconsistent predictions across consecutive frames. The left panel depicts multi-modal motion prediction. The red car represents the target agent, with its red trajectory showing the ground truth. The gray trajectories illustrate various possible future paths. The middle and right panels demonstrate missing behaviors in the predicted trajectories across consecutive time steps.
• Figure 2: Comparison of Ensembling Methods - The figure illustrates two ensembling approaches. Model Ensembling (Top): Multiple models independently predict N trajectories at Frame t. With M models, this results in M*N trajectories that are combined into the final N trajectories at the trajectory level. Temporal Ensembling (Bottom): A single model generates M*N predictions across M nearby frames. Our proposed learning-based aggregation then combines them into the final N trajectories.
• Figure 3: This highlights the precision-diversity trade-off in trajectory-level aggregation using K-means. Gray trajectories represent all trajectories within the sliding window. Orange trajectories depict single-frame approach predictions, while blue trajectories demonstrate the integration of multiple-frame predictions aggregated from the gray ones.
• Figure 4: Overall Pipeline of Temporal Ensembling with Learning-based Aggregation - The architecture consists of two main blocks. Block (a) represents the baseline model, QCNet, from which we leverage the predicted mode queries (not the final trajectories). Block (b) depicts our proposed method. It takes predictions of mode queries from nearby frames as input. Element-wise addition is used to aggregate historical mode queries. A transformer decoder then fuses the aggregated mode queries with scene embedding at time step T. Finally, each feed-forward network (FFN) predicts the final trajectory.
• Figure 5: Streaming-style formulation - Predictions exhibit a high degree of overlap in continuous datasets in the streaming-style paradigm. The overlapping time ranges, shown by the two dashed lines, offer an opportunity to exploit this property.