Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-to-End 3D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration

Zhenwei Yang, Yibo Ai, Weidong Zhang

TL;DR

The paper tackles robust 3D spatiotemporal perception in V2X-enabled autonomous driving, addressing occlusion, view limitations, and communication latency. It introduces XET-V2X, an end-to-end framework that unifies multi-view collaboration, multimodal fusion, and temporal modeling within a single architecture, utilizing dual-layer cross-attention to fuse ego and cooperative observations into a shared BEV representation. The authors demonstrate state-of-the-art performance on real-world and simulated V2X benchmarks (V2X-Seq-SPD, V2X-Sim-V2V, V2X-Sim-V2I), with substantial gains in detection and tracking under varying delays and as validated by both quantitative metrics and qualitative visualizations. The work shows significant potential for scalable, robust cooperative perception in urban driving, enabling safer and more reliable navigation through enhanced occlusion handling and longer-range understanding.

Abstract

Multi-view cooperative perception and multimodal fusion are essential for reliable 3D spatiotemporal understanding in autonomous driving, especially under occlusions, limited viewpoints, and communication delays in V2X scenarios. This paper proposes XET-V2X, a multi-modal fused end-to-end tracking framework for v2x collaboration that unifies multi-view multimodal sensing within a shared spatiotemporal representation. To efficiently align heterogeneous viewpoints and modalities, XET-V2X introduces a dual-layer spatial cross-attention module based on multi-scale deformable attention. Multi-view image features are first aggregated to enhance semantic consistency, followed by point cloud fusion guided by the updated spatial queries, enabling effective cross-modal interaction while reducing computational overhead. Experiments on the real-world V2X-Seq-SPD dataset and the simulated V2X-Sim-V2V and V2X-Sim-V2I benchmarks demonstrate consistent improvements in detection and tracking performance under varying communication delays. Both quantitative results and qualitative visualizations indicate that XET-V2X achieves robust and temporally stable perception in complex traffic scenarios.

End-to-End 3D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration

TL;DR

The paper tackles robust 3D spatiotemporal perception in V2X-enabled autonomous driving, addressing occlusion, view limitations, and communication latency. It introduces XET-V2X, an end-to-end framework that unifies multi-view collaboration, multimodal fusion, and temporal modeling within a single architecture, utilizing dual-layer cross-attention to fuse ego and cooperative observations into a shared BEV representation. The authors demonstrate state-of-the-art performance on real-world and simulated V2X benchmarks (V2X-Seq-SPD, V2X-Sim-V2V, V2X-Sim-V2I), with substantial gains in detection and tracking under varying delays and as validated by both quantitative metrics and qualitative visualizations. The work shows significant potential for scalable, robust cooperative perception in urban driving, enabling safer and more reliable navigation through enhanced occlusion handling and longer-range understanding.

Abstract

Multi-view cooperative perception and multimodal fusion are essential for reliable 3D spatiotemporal understanding in autonomous driving, especially under occlusions, limited viewpoints, and communication delays in V2X scenarios. This paper proposes XET-V2X, a multi-modal fused end-to-end tracking framework for v2x collaboration that unifies multi-view multimodal sensing within a shared spatiotemporal representation. To efficiently align heterogeneous viewpoints and modalities, XET-V2X introduces a dual-layer spatial cross-attention module based on multi-scale deformable attention. Multi-view image features are first aggregated to enhance semantic consistency, followed by point cloud fusion guided by the updated spatial queries, enabling effective cross-modal interaction while reducing computational overhead. Experiments on the real-world V2X-Seq-SPD dataset and the simulated V2X-Sim-V2V and V2X-Sim-V2I benchmarks demonstrate consistent improvements in detection and tracking performance under varying communication delays. Both quantitative results and qualitative visualizations indicate that XET-V2X achieves robust and temporally stable perception in complex traffic scenarios.
Paper Structure (38 sections, 6 equations, 6 figures, 4 tables)

This paper contains 38 sections, 6 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-View Cooperative Perception
  4. Multimodal Fusion Perception
  5. Temporal Information Fusion
  6. End-to-End Multi-Object Tracking
  7. Difference from Our Work
  8. Method
  9. Task Formulation
  10. Temporal Modeling and Notation
  11. Input Representation
  12. Output Definition
  13. Ground Truth and Training Objective
  14. Motivation and Design Rationale for Multi-View Cooperative Multimodal Fusion
  15. Model Architecture
  16. ...and 23 more sections

Figures (6)

  • Figure 1: V2X Cooperative Perception Diagram. The red vehicle denotes the ego vehicle, while the blue vehicles represent cooperative CAVs. Both the ego vehicle and cooperative CAVs are equipped with forward-facing fan-shaped cameras and LiDAR sensors with front-view coverage. The RSU is equipped with fan-shaped cameras covering the four approaches of the intersection and a $360^\circ$ LiDAR sensor, providing comprehensive infrastructure-side perception.
  • Figure 2: Architecture of XET-V2X
  • Figure 3: Backbone. (a) Point Cloud Backbone; (b) Image Backbone.
  • Figure 4: Performance comparison under different communication latency conditions across three datasets. Subfigures show: (a) V2X-Seq-SPD dataset detection performance (mAP) yu2023v2x; (b) V2X-Seq-SPD dataset tracking performance (AMOTA) yu2023v2x; (c) V2X-Seq-SPD dataset tracking performance (AMOTP) yu2023v2x; (d) V2X-Sim-V2V dataset detection performance (mAP) li2022v2x; (e) V2X-Sim-V2V dataset tracking performance (AMOTA) li2022v2x; (f) V2X-Sim-V2V dataset tracking performance (AMOTP) li2022v2x; (g) V2X-Sim-V2I dataset detection performance (mAP) li2022v2x; (h) V2X-Sim-V2I dataset tracking performance (AMOTA) li2022v2x; (i) V2X-Sim-V2I dataset tracking performance (AMOTP) li2022v2x.
  • Figure 5: Qualitative visualization and comparison of perception results on the V2X-Seq-SPD dataset yu2023v2x. (a) Qualitative visualization of XET-V2X results. (b) Qualitative comparison of different perception models.
  • ...and 1 more figures