Don't double it: Efficient Agent Prediction in Occlusions

TL;DR

MatchInformer tackles occlusion-aware agent prediction by combining a transformer-based encoder-decoder with Hungarian Matching to enforce one-to-one predictions, significantly reducing redundant occupancy hypotheses. By decoupling heading from motion and predicting a multiclass occupancy head, it improves both the plausibility and interpretability of future trajectories, while MCC provides a robust metric under class imbalance. Evaluated on the Waymo Open Motion Dataset, the method demonstrates superior occlusion reasoning and trajectory accuracy compared to SceneInformer, and achieves a compact model size (2.5M parameters) with reasonable training efficiency. The work advances occlusion-aware planning by delivering more reliable occupancy maps and multi-modal trajectory predictions, and it introduces MCC as a principled metric for imbalanced occupancy tasks.

Abstract

Occluded traffic agents pose a significant challenge for autonomous vehicles, as hidden pedestrians or vehicles can appear unexpectedly, yet this problem remains understudied. Existing learning-based methods, while capable of inferring the presence of hidden agents, often produce redundant occupancy predictions where a single agent is identified multiple times. This issue complicates downstream planning and increases computational load. To address this, we introduce MatchInformer, a novel transformer-based approach that builds on the state-of-the-art SceneInformer architecture. Our method improves upon prior work by integrating Hungarian Matching, a state-of-the-art object matching algorithm from object detection, into the training process to enforce a one-to-one correspondence between predictions and ground truth, thereby reducing redundancy. We further refine trajectory forecasts by decoupling an agent's heading from its motion, a strategy that improves the accuracy and interpretability of predicted paths. To better handle class imbalances, we propose using the Matthews Correlation Coefficient (MCC) to evaluate occupancy predictions. By considering all entries in the confusion matrix, MCC provides a robust measure even in sparse or imbalanced scenarios. Experiments on the Waymo Open Motion Dataset demonstrate that our approach improves reasoning about occluded regions and produces more accurate trajectory forecasts than prior methods.

Figures (6)

• Figure 1: We visualize our model's predictions in an occlusion scenario, contrasting them with those of the prior work, SceneInformer. As shown, SceneInformer (a) predicts a high density of occupied points, which generates numerous candidate trajectories and creates a cluttered scene for the planner. In stark contrast, our model (b) produces a much sparser and more refined prediction, demonstrating its ability to reduce redundant occupancy forecasts. In these visualizations, occupancy probability is indicated by a gradient from white (0) to deep red (1), predicted trajectories are shown in blue, and only trajectories for points with a probability greater than 0.5 are displayed.
• Figure 2: Overview of the MatchInformer training process: Key improvements over SceneInformer include Hungarian Matching before loss computation, a positional MLP that allows additional movement of the anchor points and predicts heading angles, and a class MLP that predicts occupancy and assigns each point to a class (car, pedestrian, or bicycle).
• Figure 3: Hungarian Matching is performed between predicted grid points and either the ground-truth car or the “no class” label. The point highlighted with a green outline is matched to the car, all others are assigned to “no class.” In Fig. \ref{['fig:low_cost']}, a lower class weight ($\lambda_{\text{class}} = 1.0$) matches the point directly on the car despite its low occupancy probability. In Fig. \ref{['fig:high_cost']}, a higher weight ($\lambda_{\text{class}} = 3.0$) matches a point above the car, as higher occupancy outweighs distance. Edge numbers show the matching costs to the ground-truth car.
• Figure 4: Weighted cross-entropy loss for three prediction scenarios, shown without Hungarian Matching (SceneInformer) and with Hungarian Matching (MatchInformer, our model). Since each point represents one agent, a car is matched to a single point. In case (a), the exact car position is predicted correctly. In case (b), a neighboring point is predicted as occupied, while the exact-position point has a low predicted probability. In case (c), both the exact-position point and surrounding points are predicted with high occupancy probability. Here, true positives (TP) are correctly predicted occupied points, false positives (FP) are points predicted as occupied but actually free, false negatives (FN) are missed occupied points, and true negatives (TN) are correctly predicted free points. The loss differs only in (b): without Hungarian Matching, slightly offset predictions result in both a false negative and a false positive, which significantly increases the loss. With Hungarian Matching, the offset prediction is paired with the ground-truth object, incurring only a small additional positional cost.
• Figure 5: Comparison of the occupancy prediction without Hungarian Matching (SceneInformer) and with Hungarian Matching (MatchInformer). In both scenarios, a car (blue) was previously observed when not occluded, and a car (green) was unobserved. In Scenario 1 (a), both models fail to predict the unobserved car. In Scenario 2 (b), SceneInformer predicts occupancy over a large area to actively avoid missing the ground-truth agent, while MatchInformer predicts only a few points within the region, capturing uncertainty without overprediction. This also applies to the previously observed car, where reasoning about speed, heading, and acceleration is required to infer its current position.