Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FipTR: A Simple yet Effective Transformer Framework for Future Instance Prediction in Autonomous Driving

Xingtai Gui, Tengteng Huang, Haonan Shao, Haotian Yao, Chi Zhang

TL;DR

This paper addresses the challenge of BEV future instance prediction for autonomous driving by proposing FipTR, a fully end-to-end transformer framework that directly outputs future occupancy masks for traffic participants through instance queries. It introduces a Flow-aware BEV Predictor with backward flow guided deformable attention to evolve BEV features over time and a Future Instance Matching strategy to ensure consistent instance IDs across multiple future frames. Key contributions include the end-to-end architecture, the flow-guided sampling mechanism, and multi-frame multi-task matching that eliminates heavy post-processing while improving temporal coherence, validated by state-of-the-art VPQ and IoU on NuScenes across BEV encoders. The work promises simpler, more interpretable BEV forecasting with practical impact on planning and safety in autonomous driving, and the released code enables reproducibility and further research.

Abstract

The future instance prediction from a Bird's Eye View(BEV) perspective is a vital component in autonomous driving, which involves future instance segmentation and instance motion prediction. Existing methods usually rely on a redundant and complex pipeline which requires multiple auxiliary outputs and post-processing procedures. Moreover, estimated errors on each of the auxiliary predictions will lead to degradation of the prediction performance. In this paper, we propose a simple yet effective fully end-to-end framework named Future Instance Prediction Transformer(FipTR), which views the task as BEV instance segmentation and prediction for future frames. We propose to adopt instance queries representing specific traffic participants to directly estimate the corresponding future occupied masks, and thus get rid of complex post-processing procedures. Besides, we devise a flow-aware BEV predictor for future BEV feature prediction composed of a flow-aware deformable attention that takes backward flow guiding the offset sampling. A novel future instance matching strategy is also proposed to further improve the temporal coherence. Extensive experiments demonstrate the superiority of FipTR and its effectiveness under different temporal BEV encoders. The code is available at https://github.com/TabGuigui/FipTR .

FipTR: A Simple yet Effective Transformer Framework for Future Instance Prediction in Autonomous Driving

TL;DR

This paper addresses the challenge of BEV future instance prediction for autonomous driving by proposing FipTR, a fully end-to-end transformer framework that directly outputs future occupancy masks for traffic participants through instance queries. It introduces a Flow-aware BEV Predictor with backward flow guided deformable attention to evolve BEV features over time and a Future Instance Matching strategy to ensure consistent instance IDs across multiple future frames. Key contributions include the end-to-end architecture, the flow-guided sampling mechanism, and multi-frame multi-task matching that eliminates heavy post-processing while improving temporal coherence, validated by state-of-the-art VPQ and IoU on NuScenes across BEV encoders. The work promises simpler, more interpretable BEV forecasting with practical impact on planning and safety in autonomous driving, and the released code enables reproducibility and further research.

Abstract

The future instance prediction from a Bird's Eye View(BEV) perspective is a vital component in autonomous driving, which involves future instance segmentation and instance motion prediction. Existing methods usually rely on a redundant and complex pipeline which requires multiple auxiliary outputs and post-processing procedures. Moreover, estimated errors on each of the auxiliary predictions will lead to degradation of the prediction performance. In this paper, we propose a simple yet effective fully end-to-end framework named Future Instance Prediction Transformer(FipTR), which views the task as BEV instance segmentation and prediction for future frames. We propose to adopt instance queries representing specific traffic participants to directly estimate the corresponding future occupied masks, and thus get rid of complex post-processing procedures. Besides, we devise a flow-aware BEV predictor for future BEV feature prediction composed of a flow-aware deformable attention that takes backward flow guiding the offset sampling. A novel future instance matching strategy is also proposed to further improve the temporal coherence. Extensive experiments demonstrate the superiority of FipTR and its effectiveness under different temporal BEV encoders. The code is available at https://github.com/TabGuigui/FipTR .
Paper Structure (19 sections, 9 equations, 3 figures, 6 tables)

This paper contains 19 sections, 9 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. BEV Perception
  4. Segmentation Transformer
  5. Future Instance Prediction
  6. Methodology
  7. Overview Architecture
  8. Temporal BEV Encoder
  9. Flow-aware BEV Predictor
  10. Instance Masks Decoder
  11. Future Instance Matching
  12. Overall loss function
  13. Experiment
  14. Datasets and settings
  15. Metrics
  16. ...and 4 more sections

Figures (3)

  • Figure 1: The overall architecture of FipTR. Given the multi-frame surrounding images, a temporal BEV Encoder including image backbone, view transformer and temporal fusion module, generates the current frame BEV feature map $F_0$. The flow-aware BEV predictor takes $F_0$ and unified future BEV query as input and updates the future BEV feature map iteratively. The flow-aware deformable attention(FADA) takes the predicted backward flow into consideration and generates motion-aware sampling offsets for BEV prediction. The future masks decoder outputs the future instance segmentation by conducting element-wise product between predicted BEV feature sequence and instance queries and predicting the corresponding category and 3D box attributes.
  • Figure 2: Visualization of ground truth backward flow and mean of sampling offsets at different timestamps. The color represents the direction and the transparency represents the magnitude.
  • Figure 3: Demonstration of performance of FipTR compared with BEVerse and Ground Truth. The left part is the detection performance and only vehicles are considered. The right part is the segmentation performance where darker regions are the result of the current frame while the lighter ones are the prediction in the future frames.