MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction

TL;DR

MultiPath++ advances autonomous driving behavior prediction by shifting from dense raster encodings to sparse, polyline- and state-based representations, enabling scalable modeling of long-range interactions. It introduces multi-context gating to fuse heterogeneous inputs, learns latent anchor embeddings to replace static anchors, and leverages ensemble-based aggregation to capture diverse futures in a probabilistic GMM output. Across Waymo Open Motion Dataset and Argoverse benchmarks, the approach achieves state-of-the-art or top-tier performance, driven by ablations that validate sparse encodings, learned anchors, and robust ensembling. The work delivers practical architectural guidance for efficient, diverse, and reliable trajectory prediction in real-world driving scenarios.

Abstract

Predicting the future behavior of road users is one of the most challenging and important problems in autonomous driving. Applying deep learning to this problem requires fusing heterogeneous world state in the form of rich perception signals and map information, and inferring highly multi-modal distributions over possible futures. In this paper, we present MultiPath++, a future prediction model that achieves state-of-the-art performance on popular benchmarks. MultiPath++ improves the MultiPath architecture by revisiting many design choices. The first key design difference is a departure from dense image-based encoding of the input world state in favor of a sparse encoding of heterogeneous scene elements: MultiPath++ consumes compact and efficient polylines to describe road features, and raw agent state information directly (e.g., position, velocity, acceleration). We propose a context-aware fusion of these elements and develop a reusable multi-context gating fusion component. Second, we reconsider the choice of pre-defined, static anchors, and develop a way to learn latent anchor embeddings end-to-end in the model. Lastly, we explore ensembling and output aggregation techniques -- common in other ML domains -- and find effective variants for our probabilistic multimodal output representation. We perform an extensive ablation on these design choices, and show that our proposed model achieves state-of-the-art performance on the Argoverse Motion Forecasting Competition and the Waymo Open Dataset Motion Prediction Challenge.

Figures (8)

• Figure 1: MultiPath++ Model Architecture. MCG denotes Multi-Context Gating, described in Section \ref{['sec:model']}. Blocks in red highlight portions of the model with learned parameters. Dotted inputs to the MCG denotes context features. Each of the encoder MCG outputs aggregated embeddings (one per agent) as shown by dotted arrows. On the other hand, the predictor MCG outputs one embedding per trajectory per agent
• Figure 2: Left: Context gating (CG) block diagram. Right: 3 CG blocks stacked together, with running-average skip-connections (shown as components labeled "$\mu$"). See Section \ref{['sec:mcg']} for details.
• Figure 3: A comparison of the element relationship graph for cross-attention and CG. In cross-attention, each element $\mathbf{s}_i$ aggregates information from $\mathbf{c}_{1:m}$. In CG, $\mathbf{c}_{1:m}$ summarized with a single context vector $\textbf{c}$.
• Figure 4: Examples of MultiPath++ predictions for 8 seconds in WOMD scenes. Hue indicates time horizon while transparency indicates predicted probability. Rectangles indicate vehicles while small squares indicate pedestrians. (a): A four-way intersection involving multiple interactions. For example, car A is predicted to yield for car B. (b & c): Narrow road interaction. Car A is predicted to yield for car B and then nudge around the parking car, designated by the arrow. (d & e): Interaction between two vehicles at an intersection where can A is predicted to yield for car B or make a right-turn behind. After car B passes, car A can make a left turn. Also, note the bimodal prediction of the pedestrian that is located at the corner. (f and g): Predictions in a parking lot and atypical roadgraph.
• Figure 5: Examples of MultiPath++ predictions for 8 seconds in Argoverse scenes, showing the ability to follow different lane geometries.