Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AYDIV: Adaptable Yielding 3D Object Detection via Integrated Contextual Vision Transformer

Tanmoy Dam, Sanjay Bhargav Dharavath, Sameer Alam, Nimrod Lilith, Supriyo Chakraborty, Mir Feroskhan

TL;DR

AYDIV tackles long-range 3D object detection by fusing LiDAR and camera data through a tri-phase transformer-based fusion: GCFAT provides depth-informed image features, SFFA enables sparse, depth-guided LiDAR–image alignment, and VGA fuses RoI-level voxel and image features in a volumetric grid. The framework demonstrates strong performance on Waymo Open Dataset and Argoverse2, achieving 82.04 mAPH (L2) on Waymo and a 7.40% AP improvement on AV2, outperforming several fusion-based baselines without heavy ensembling or test-time augmentation. Ablation studies confirm the necessity of each component and the global depth query for robust cross-modal fusion. These results indicate AYDIV’s potential to improve safety-critical perception in autonomous systems and motivate future work on robustness and broader domain applications.

Abstract

Combining LiDAR and camera data has shown potential in enhancing short-distance object detection in autonomous driving systems. Yet, the fusion encounters difficulties with extended distance detection due to the contrast between LiDAR's sparse data and the dense resolution of cameras. Besides, discrepancies in the two data representations further complicate fusion methods. We introduce AYDIV, a novel framework integrating a tri-phase alignment process specifically designed to enhance long-distance detection even amidst data discrepancies. AYDIV consists of the Global Contextual Fusion Alignment Transformer (GCFAT), which improves the extraction of camera features and provides a deeper understanding of large-scale patterns; the Sparse Fused Feature Attention (SFFA), which fine-tunes the fusion of LiDAR and camera details; and the Volumetric Grid Attention (VGA) for a comprehensive spatial data fusion. AYDIV's performance on the Waymo Open Dataset (WOD) with an improvement of 1.24% in mAPH value(L2 difficulty) and the Argoverse2 Dataset with a performance improvement of 7.40% in AP value demonstrates its efficacy in comparison to other existing fusion-based methods. Our code is publicly available at https://github.com/sanjay-810/AYDIV2

AYDIV: Adaptable Yielding 3D Object Detection via Integrated Contextual Vision Transformer

TL;DR

AYDIV tackles long-range 3D object detection by fusing LiDAR and camera data through a tri-phase transformer-based fusion: GCFAT provides depth-informed image features, SFFA enables sparse, depth-guided LiDAR–image alignment, and VGA fuses RoI-level voxel and image features in a volumetric grid. The framework demonstrates strong performance on Waymo Open Dataset and Argoverse2, achieving 82.04 mAPH (L2) on Waymo and a 7.40% AP improvement on AV2, outperforming several fusion-based baselines without heavy ensembling or test-time augmentation. Ablation studies confirm the necessity of each component and the global depth query for robust cross-modal fusion. These results indicate AYDIV’s potential to improve safety-critical perception in autonomous systems and motivate future work on robustness and broader domain applications.

Abstract

Combining LiDAR and camera data has shown potential in enhancing short-distance object detection in autonomous driving systems. Yet, the fusion encounters difficulties with extended distance detection due to the contrast between LiDAR's sparse data and the dense resolution of cameras. Besides, discrepancies in the two data representations further complicate fusion methods. We introduce AYDIV, a novel framework integrating a tri-phase alignment process specifically designed to enhance long-distance detection even amidst data discrepancies. AYDIV consists of the Global Contextual Fusion Alignment Transformer (GCFAT), which improves the extraction of camera features and provides a deeper understanding of large-scale patterns; the Sparse Fused Feature Attention (SFFA), which fine-tunes the fusion of LiDAR and camera details; and the Volumetric Grid Attention (VGA) for a comprehensive spatial data fusion. AYDIV's performance on the Waymo Open Dataset (WOD) with an improvement of 1.24% in mAPH value(L2 difficulty) and the Argoverse2 Dataset with a performance improvement of 7.40% in AP value demonstrates its efficacy in comparison to other existing fusion-based methods. Our code is publicly available at https://github.com/sanjay-810/AYDIV2
Paper Structure (14 sections, 4 equations, 5 figures, 5 tables)

This paper contains 14 sections, 4 equations, 5 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. AYDIV
  4. LiDAR Feature Extraction through Voxelization
  5. GCFAT
  6. SFFA
  7. VGA
  8. Loss function
  9. Experiments
  10. Dataset details
  11. Implementation Details
  12. Performance on WOD and AV2
  13. Ablation Studies on WOD
  14. CONCLUSIONS

Figures (5)

  • Figure 1: AYDIV Pipeline: The pipeline integrates multiple modalities through three essential parts: GCFAT, SFFA, and VGA. Each of these parts plays an important role in the complex interaction of local and global contextual attention mechanisms. GCFAT considers the camera as a local feature and the depth information as a global query through its image and LiDAR point clouds. SFFA aligns the image features extracted by GCFAT with the voxelized LiDAR features in a cohesive feature alignment manner. Finally, VGA combines sparse voxelized LiDAR features and SFFA camera features in a grid space.
  • Figure 2: GCFAT Comprises LMSA and GDA
  • Figure 3: SFFA combines extracted LiDAR and image features.
  • Figure 4: VGA: Volumetric integration between $\mathbb{F}_i^{LiDAR}$ and $\mathbb{F}_i^{SFFA}$ though $i^{th}$ channel.
  • Figure 5: A qualitative comparison of long-range 3D object detection. BEV maps on left, 2D image from camera 1 on right. Red: Predicted boxes. Black: Detection score.