Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Li-ViP3D++: Query-Gated Deformable Camera-LiDAR Fusion for End-to-End Perception and Trajectory Prediction

Matej Halinkovic, Nina Masarykova, Alexey Vinel, Marek Galinski

TL;DR

Li-ViP3D++ addresses robust end-to-end perception and trajectory prediction by fusing camera and LiDAR information in the query space through a differentiable Query-Gated Deformable Fusion mechanism. By sampling image and LiDAR features per agent query and gating their contributions, the model jointly optimizes detection, tracking, and multi-hypothesis forecasting. On nuScenes, it achieves higher EPA and mAP with significantly fewer false positives and favorable latency compared with prior Li-ViP3D and ViP3D baselines, demonstrating improved robustness and deployability. This work demonstrates the value of query-space multimodal fusion for end-to-end autonomous driving perception pipelines and points to future reductions in map dependence and improvements under adverse conditions.

Abstract

End-to-end perception and trajectory prediction from raw sensor data is one of the key capabilities for autonomous driving. Modular pipelines restrict information flow and can amplify upstream errors. Recent query-based, fully differentiable perception-and-prediction (PnP) models mitigate these issues, yet the complementarity of cameras and LiDAR in the query-space has not been sufficiently explored. Models often rely on fusion schemes that introduce heuristic alignment and discrete selection steps which prevent full utilization of available information and can introduce unwanted bias. We propose Li-ViP3D++, a query-based multimodal PnP framework that introduces Query-Gated Deformable Fusion (QGDF) to integrate multi-view RGB and LiDAR in query space. QGDF (i) aggregates image evidence via masked attention across cameras and feature levels, (ii) extracts LiDAR context through fully differentiable BEV sampling with learned per-query offsets, and (iii) applies query-conditioned gating to adaptively weight visual and geometric cues per agent. The resulting architecture jointly optimizes detection, tracking, and multi-hypothesis trajectory forecasting in a single end-to-end model. On nuScenes, Li-ViP3D++ improves end-to-end behavior and detection quality, achieving higher EPA (0.335) and mAP (0.502) while substantially reducing false positives (FP ratio 0.147), and it is faster than the prior Li-ViP3D variant (139.82 ms vs. 145.91 ms). These results indicate that query-space, fully differentiable camera-LiDAR fusion can increase robustness of end-to-end PnP without sacrificing deployability.

Li-ViP3D++: Query-Gated Deformable Camera-LiDAR Fusion for End-to-End Perception and Trajectory Prediction

TL;DR

Li-ViP3D++ addresses robust end-to-end perception and trajectory prediction by fusing camera and LiDAR information in the query space through a differentiable Query-Gated Deformable Fusion mechanism. By sampling image and LiDAR features per agent query and gating their contributions, the model jointly optimizes detection, tracking, and multi-hypothesis forecasting. On nuScenes, it achieves higher EPA and mAP with significantly fewer false positives and favorable latency compared with prior Li-ViP3D and ViP3D baselines, demonstrating improved robustness and deployability. This work demonstrates the value of query-space multimodal fusion for end-to-end autonomous driving perception pipelines and points to future reductions in map dependence and improvements under adverse conditions.

Abstract

End-to-end perception and trajectory prediction from raw sensor data is one of the key capabilities for autonomous driving. Modular pipelines restrict information flow and can amplify upstream errors. Recent query-based, fully differentiable perception-and-prediction (PnP) models mitigate these issues, yet the complementarity of cameras and LiDAR in the query-space has not been sufficiently explored. Models often rely on fusion schemes that introduce heuristic alignment and discrete selection steps which prevent full utilization of available information and can introduce unwanted bias. We propose Li-ViP3D++, a query-based multimodal PnP framework that introduces Query-Gated Deformable Fusion (QGDF) to integrate multi-view RGB and LiDAR in query space. QGDF (i) aggregates image evidence via masked attention across cameras and feature levels, (ii) extracts LiDAR context through fully differentiable BEV sampling with learned per-query offsets, and (iii) applies query-conditioned gating to adaptively weight visual and geometric cues per agent. The resulting architecture jointly optimizes detection, tracking, and multi-hypothesis trajectory forecasting in a single end-to-end model. On nuScenes, Li-ViP3D++ improves end-to-end behavior and detection quality, achieving higher EPA (0.335) and mAP (0.502) while substantially reducing false positives (FP ratio 0.147), and it is faster than the prior Li-ViP3D variant (139.82 ms vs. 145.91 ms). These results indicate that query-space, fully differentiable camera-LiDAR fusion can increase robustness of end-to-end PnP without sacrificing deployability.
Paper Structure (19 sections, 22 equations, 4 figures, 2 tables)

This paper contains 19 sections, 22 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Perception and Prediction Pipelines
  4. End-to-End solutions
  5. Attention-based RGB and LiDAR fusion
  6. Methodology
  7. Feature extraction
  8. Modality fusion
  9. Image feature aggregation via masked attention over cameras and pyramid levels
  10. LiDAR feature aggregation via differentiable BEV sampling
  11. Query-aware gated multimodal fusion
  12. Residual update with positional encoding
  13. Agent Perception and Prediction
  14. Results
  15. Discussion
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overall architecture of the Li-ViP3D++ multimodal end-to-end trajectory prediction model. The QGDF module is our primary contribution which handles sampling and fusion to ensure efficient usage of information from each input modality.
  • Figure 2: Diagram of the Query-Gated Deformable Fusion (QGDF) mechanism and sampling from each modality.
  • Figure 3: Visual comparisons of the predicted trajectories for each of the compared agent query-based models. Each model predicted K=6 possible future trajectories, we visualize the trajectory with the highest confidence for each sample.
  • Figure 4: Average weight of RGB and LiDAR and modalities across all refinement layers of Li-ViP3D on testing data.