SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction

TL;DR

This paper introduces a novel scenario-adaptive refinement strategy, named SmartRefine, to refine prediction with minimal additional computation, and shows that the method consistently improves the prediction accuracy of multiple state-of-the-art prediction models.

Abstract

Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed environments. Context information, such as road maps and surrounding agents' states, provides crucial geometric and semantic information for motion behavior prediction. To this end, recent works explore two-stage prediction frameworks where coarse trajectories are first proposed, and then used to select critical context information for trajectory refinement. However, they either incur a large amount of computation or bring limited improvement, if not both. In this paper, we introduce a novel scenario-adaptive refinement strategy, named SmartRefine, to refine prediction with minimal additional computation. Specifically, SmartRefine can comprehensively adapt refinement configurations based on each scenario's properties, and smartly chooses the number of refinement iterations by introducing a quality score to measure the prediction quality and remaining refinement potential of each scenario. SmartRefine is designed as a generic and flexible approach that can be seamlessly integrated into most state-of-the-art motion prediction models. Experiments on Argoverse (1 & 2) show that our method consistently improves the prediction accuracy of multiple state-of-the-art prediction models. Specifically, by adding SmartRefine to QCNet, we outperform all published ensemble-free works on the Argoverse 2 leaderboard (single agent track) at submission. Comprehensive studies are also conducted to ablate design choices and explore the mechanism behind multi-iteration refinement. Codes are available at https://github.com/opendilab/SmartRefine/

Figures (15)

• Figure 1: Overview of our framework. The top section concisely illustrates the full pipeline, while the lower section introduces details of three core modules. We first pass HD map and agent information to a prediction model backbone, generating initial trajectories and trajectory features. The initial predicted trajectory is then used to adaptively select anchors and retrieve critical context elements (bottom left). The contexts retrieved by each anchor are transformed to the coordinate frame centered at the corresponding anchors (bottom middle). The encoded contexts are then utilized to refine each trajectory segment by generating the offset of each trajectory segment (bottom right). Our model will predict a quality score measuring the prediction quality, and adaptively decide the number of refinement iterations (upper right). Our method is lightweight and can be seamlessly integrated with most existing motion prediction models.
• Figure 2: Comparison between the fixed and adaptive number of refinement iterations. For the adaptive methods, We tested different quality score thresholds $\Bar{q}$ mentioned in Algorithm \ref{['algo:inference']}.
• Figure 3: A study to understand the mechanism behind the refinement. Specifically, We mark the quality score distribution of the predictive trajectory before refinement, and track how the quality score changes along the multi-iteration refinement. We can see while the overall performance is improved, not every trajectory benefits from refinement, which implies the necessity of adaptive refinement. See Sec. \ref{['sec:discussion']} for detailed discussions.
• Figure 4: Comparison between the fixed and adaptive number of refinement iterations, when we apply our SmartRefine on all six considered backbones respectively (on val set of Argoverse and Argoverse 2). The blue curve represents fixed refinement iterations (for both training and inference). Other curves denote adaptive refinement iterations with different quality score threshold $\Bar{q}$ during inference time (5 refinement iterations are utilized during training). For each threshold, we ablate different limits for maximum refinement iteration $I'$, resulting in 5 points for each curve (refer to Algorithm \ref{['algo:inference']} for detailed descriptions of $\Bar{q}$ and $I'$). Two observations: 1) on HiVT, ProphNet, DenseTNT, and QCNet (no ref), our adaptive refinement strategy outperforms the fixed refinement strategy given any threshold $\bar{q}$. 2) on mmTransformer, and QCNet, our adaptive refinement strategy outperforms the fixed refinement strategy when we set a higher threshold $\bar{q}$ (0.4, 0.5, 0.6) when we decide whether another refinement iteration is needed.
• Figure 5: HiVT w/ Ours in Argoverse