Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kaninfradet3D:A Road-side Camera-LiDAR Fusion 3D Perception Model based on Nonlinear Feature Extraction and Intrinsic Correlation

Pei Liu, Nanfang Zheng, Yiqun Li, Junlan Chen, Ziyuan Pu

TL;DR

Kaninfradet3D is proposed, which optimizes the feature extraction and fusion modules and indicates that Kaninfradet3D can effectively fuse features, demonstrating the potential of applying KANs in roadside perception tasks.

Abstract

With the development of AI-assisted driving, numerous methods have emerged for ego-vehicle 3D perception tasks, but there has been limited research on roadside perception. With its ability to provide a global view and a broader sensing range, the roadside perspective is worth developing. LiDAR provides precise three-dimensional spatial information, while cameras offer semantic information. These two modalities are complementary in 3D detection. However, adding camera data does not increase accuracy in some studies since the information extraction and fusion procedure is not sufficiently reliable. Recently, Kolmogorov-Arnold Networks (KANs) have been proposed as replacements for MLPs, which are better suited for high-dimensional, complex data. Both the camera and the LiDAR provide high-dimensional information, and employing KANs should enhance the extraction of valuable features to produce better fusion outcomes. This paper proposes Kaninfradet3D, which optimizes the feature extraction and fusion modules. To extract features from complex high-dimensional data, the model's encoder and fuser modules were improved using KAN Layers. Cross-attention was applied to enhance feature fusion, and visual comparisons verified that camera features were more evenly integrated. This addressed the issue of camera features being abnormally concentrated, negatively impacting fusion. Compared to the benchmark, our approach shows improvements of +9.87 mAP and +10.64 mAP in the two viewpoints of the TUMTraf Intersection Dataset and an improvement of +1.40 mAP in the roadside end of the TUMTraf V2X Cooperative Perception Dataset. The results indicate that Kaninfradet3D can effectively fuse features, demonstrating the potential of applying KANs in roadside perception tasks.

Kaninfradet3D:A Road-side Camera-LiDAR Fusion 3D Perception Model based on Nonlinear Feature Extraction and Intrinsic Correlation

TL;DR

Kaninfradet3D is proposed, which optimizes the feature extraction and fusion modules and indicates that Kaninfradet3D can effectively fuse features, demonstrating the potential of applying KANs in roadside perception tasks.

Abstract

With the development of AI-assisted driving, numerous methods have emerged for ego-vehicle 3D perception tasks, but there has been limited research on roadside perception. With its ability to provide a global view and a broader sensing range, the roadside perspective is worth developing. LiDAR provides precise three-dimensional spatial information, while cameras offer semantic information. These two modalities are complementary in 3D detection. However, adding camera data does not increase accuracy in some studies since the information extraction and fusion procedure is not sufficiently reliable. Recently, Kolmogorov-Arnold Networks (KANs) have been proposed as replacements for MLPs, which are better suited for high-dimensional, complex data. Both the camera and the LiDAR provide high-dimensional information, and employing KANs should enhance the extraction of valuable features to produce better fusion outcomes. This paper proposes Kaninfradet3D, which optimizes the feature extraction and fusion modules. To extract features from complex high-dimensional data, the model's encoder and fuser modules were improved using KAN Layers. Cross-attention was applied to enhance feature fusion, and visual comparisons verified that camera features were more evenly integrated. This addressed the issue of camera features being abnormally concentrated, negatively impacting fusion. Compared to the benchmark, our approach shows improvements of +9.87 mAP and +10.64 mAP in the two viewpoints of the TUMTraf Intersection Dataset and an improvement of +1.40 mAP in the roadside end of the TUMTraf V2X Cooperative Perception Dataset. The results indicate that Kaninfradet3D can effectively fuse features, demonstrating the potential of applying KANs in roadside perception tasks.

Paper Structure

This paper contains 17 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Roadside perception
  4. Kolmogorov–Arnold Networks (KANs)
  5. Multi-sensor Fusion
  6. Methodology
  7. KAN-Improved Feature Extraction Module
  8. KANLayers
  9. Kolmogorov-Arnold Convolutions(KANConv)
  10. KANvtransform, PointEncoder and ConKANfuser
  11. Camera LiDAR Cross-Attention Block
  12. Experiments and Result Analysis
  13. Experimental settings
  14. Results
  15. Ablation Study on Camera-LiDAR Attention
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Model Structure of Kaninfradetr3D
  • Figure 2: Visual comparison between LiDAR feature and Camera feature.The red-circled elements of the same target are easily identifiable in LiDAR, but the camera feature do not clearly separate the target from the backdrop.
  • Figure 3: Comparison of features after fusion of Convfuser and Camera-LiDAR CrossAttn. The part circled in red is the abnormally fused camera feature. The gray background image is a version with the background removed for clearer display.