Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention

Harsh Yadav, Maximilian Schaefer, Kun Zhao, Tobias Meisen

TL;DR

CASPFormer addresses the scalability challenge of HD-map dependent motion prediction by enabling multi-modal trajectory prediction from rasterized BEV images. It introduces a deformable-attention-based recurrent decoder that uses two learnable embeddings—temporal and mode queries—to capture temporal structure and prevent mode collapse, producing diverse, scene-consistent vectorized trajectories. Evaluated on nuScenes, CASPFormer achieves state-of-the-art performance on key metrics and demonstrates robust multi-modal behavior, with qualitative trajectories reflecting different ego speeds. The work offers a scalable, perception-module-compatible approach for autonomous driving that reduces reliance on costly HD maps and has practical implications for real-time trajectory forecasting and planning.

Abstract

Motion prediction is an important aspect for Autonomous Driving (AD) and Advance Driver Assistance Systems (ADAS). Current state-of-the-art motion prediction methods rely on High Definition (HD) maps for capturing the surrounding context of the ego vehicle. Such systems lack scalability in real-world deployment as HD maps are expensive to produce and update in real-time. To overcome this issue, we propose Context Aware Scene Prediction Transformer (CASPFormer), which can perform multi-modal motion prediction from rasterized Bird-Eye-View (BEV) images. Our system can be integrated with any upstream perception module that is capable of generating BEV images. Moreover, CASPFormer directly decodes vectorized trajectories without any postprocessing. Trajectories are decoded recurrently using deformable attention, as it is computationally efficient and provides the network with the ability to focus its attention on the important spatial locations of the BEV images. In addition, we also address the issue of mode collapse for generating multiple scene-consistent trajectories by incorporating learnable mode queries. We evaluate our model on the nuScenes dataset and show that it reaches state-of-the-art across multiple metrics

CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention

TL;DR

CASPFormer addresses the scalability challenge of HD-map dependent motion prediction by enabling multi-modal trajectory prediction from rasterized BEV images. It introduces a deformable-attention-based recurrent decoder that uses two learnable embeddings—temporal and mode queries—to capture temporal structure and prevent mode collapse, producing diverse, scene-consistent vectorized trajectories. Evaluated on nuScenes, CASPFormer achieves state-of-the-art performance on key metrics and demonstrates robust multi-modal behavior, with qualitative trajectories reflecting different ego speeds. The work offers a scalable, perception-module-compatible approach for autonomous driving that reduces reliance on costly HD maps and has practical implications for real-time trajectory forecasting and planning.

Abstract

Motion prediction is an important aspect for Autonomous Driving (AD) and Advance Driver Assistance Systems (ADAS). Current state-of-the-art motion prediction methods rely on High Definition (HD) maps for capturing the surrounding context of the ego vehicle. Such systems lack scalability in real-world deployment as HD maps are expensive to produce and update in real-time. To overcome this issue, we propose Context Aware Scene Prediction Transformer (CASPFormer), which can perform multi-modal motion prediction from rasterized Bird-Eye-View (BEV) images. Our system can be integrated with any upstream perception module that is capable of generating BEV images. Moreover, CASPFormer directly decodes vectorized trajectories without any postprocessing. Trajectories are decoded recurrently using deformable attention, as it is computationally efficient and provides the network with the ability to focus its attention on the important spatial locations of the BEV images. In addition, we also address the issue of mode collapse for generating multiple scene-consistent trajectories by incorporating learnable mode queries. We evaluate our model on the nuScenes dataset and show that it reaches state-of-the-art across multiple metrics
Paper Structure (14 sections, 6 equations, 5 figures, 2 tables)

This paper contains 14 sections, 6 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Input Scene Representation
  4. Multi-Modal Prediction
  5. Transformer-based Attention in Image Domain
  6. Methods
  7. Input-Output Formulation
  8. Network Architecture
  9. Loss Formulation
  10. Experiments
  11. Experimental Setup
  12. Results
  13. Ablation Studies
  14. Conclusion

Figures (5)

  • Figure 1: Shows an overview of the CASPFormer architecture. The backbone uses CNN and convolution RNN to generate the scene encodings. The scene encodings have a pyramid structure with increasing resolution from top to bottom. The deformable self-attention module applies a multi-scale feature fusion on the scene encodings, while the deformable cross-attention module recurrently decodes the trajectories. The output of the previous time step is used to update the position of the reference point and the query embeddings in deformable cross-attention.
  • Figure 2: A depiction of the recurrent decoder network architecture. The position embeddings are non-learnable and help the network in learning the location of features. The mode queries serve the purpose of producing multiple scene-consistent trajectories in the multi-modal output. The temporal correlation in the predicted trajectories is captured with temporal queries. The position of the reference point for the deformable attention is set to the ego vehicle position in the scene. The recurrent architecture updates the ego vehicle position and the temporal queries at every recurrent step.
  • Figure 3: Left column shows the predicted trajectories by the network without mode queries. The right column shows the corresponding scenarios after the mode queries are incorporated into the network. The generalization capability of the network improves with mode queries as the network can predict the trajectories that can follow multiple scene-consistent paths.
  • Figure 4: An illustration of the proposed deformable cross-attention module. The offsets in the deformable cross-attention layer are computed with the linear transformation of the queries (as is done in original deformable attention zhu2021deformable). These queries are generated by summing up temporal queries and mode queries. Values are then sampled from the multi-scale fused scene encodings at these offset locations and a weighted sum of the sampled values is computed. This process is repeated N times to produce the output queries.
  • Figure 5: Qualitative results on nuScenes prediction validation split. The blue region represents the drivable area and the green overlays portray the driving lanes. The pedestrian crossing is shown in red color. The current position of the ego vehicle is indicated with the black dot at the end of the input trajectory. The network can predict multiple scene-consistent trajectories in diverse scenarios such as intersections and crossings.