LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation

TL;DR

LiteFusion reframes LiDAR as a geometric cue instead of a separate 3D backbone, introducing a quaternion-based cross-modal embedding and a LiDAR Geometry Integrator that progressively injects depth and geometry into a camera-based 3D detector. This minimal-modification approach achieves state-of-the-art-like gains with only about 1% additional parameters and remains effective when LiDAR data is unavailable, demonstrating strong deployment versatility. The method delivers robust improvements across diverse vision backbones on nuScenes, and ablations confirm the complementary benefits of Depth-Aware Embedding and Geometry-Aware Embedding within a progressive fusion framework. Overall, LiteFusion offers a practical, scalable pathway to robust multi-modal 3D perception without the engineering burden of 3D sparse backbones.

Abstract

3D object detection is fundamental for safe and robust intelligent transportation systems. Current multi-modal 3D object detectors often rely on complex architectures and training strategies to achieve higher detection accuracy. However, these methods heavily rely on the LiDAR sensor so that they suffer from large performance drops when LiDAR is absent, which compromises the robustness and safety of autonomous systems in practical scenarios. Moreover, existing multi-modal detectors face difficulties in deployment on diverse hardware platforms, such as NPUs and FPGAs, due to their reliance on 3D sparse convolution operators, which are primarily optimized for NVIDIA GPUs. To address these challenges, we reconsider the role of LiDAR in the camera-LiDAR fusion paradigm and introduce a novel multi-modal 3D detector, LiteFusion. Instead of treating LiDAR point clouds as an independent modality with a separate feature extraction backbone, LiteFusion utilizes LiDAR data as a complementary source of geometric information to enhance camera-based detection. This straightforward approach completely eliminates the reliance on a 3D backbone, making the method highly deployment-friendly. Specifically, LiteFusion integrates complementary features from LiDAR points into image features within a quaternion space, where the orthogonal constraints are well-preserved during network training. This helps model domain-specific relations across modalities, yielding a compact cross-modal embedding. Experiments on the nuScenes dataset show that LiteFusion improves the baseline vision-based detector by +20.4% mAP and +19.7% NDS with a minimal increase in parameters (1.1%) without using dedicated LiDAR encoders. Notably, even in the absence of LiDAR input, LiteFusion maintains strong results , highlighting its favorable robustness and effectiveness across diverse fusion paradigms and deployment scenarios.

Figures (9)

• Figure 1: Illustration of the camera-LiDAR fusion paradigms. Existing multi-sensor fusion methods often rely on large 2D and 3D backbones, complex modality interactions, and elaborate training stages. In contrast, our approach simplifies the pipeline by using a progressive modality interaction strategy, eliminating the need for a 3D feature extractor. By adding only 1.1% more parameters to the camera-based framework, we achieve a significant boost in 3D perception performance, with improvements of +20.4% in mAP and +19.7% in NDS.
• Figure 2: Overall framework of the proposed approach. LiteFusion leverages the camera-based LiDAR-assisted fusion scheme, taming a camera-based 3D object detector to a multi-modal detector via the devised LiDAR geometry integrator, where the proposed DAE and GAE modules efficiently generate LiDAR-derived geometric information from the PV and BEV perspectives separately. These LiDAR features are hierarchically forwarded to the vision-based detector to enhance the image feature, progressively bolstering the 3D spatial awareness and performance of the detector.
• Figure 3: Detailed design of the proposed DAE. (a) DAE aligns the depth-aware feature dimensions with camera features, followed by a convolution operation to streamline the latent space dimensions. The two data streams are then concatenated and projected into suprasphere space using a quaternion layer for effective fused feature representation. Key computational steps of the quaternion network are illustrated in (b). Finally, the enhanced features are mapped back to their original dimensions.
• Figure 4: Detailed design of the proposed GAE. The input streams are combined through the concatenation operation, followed by a convolution operation to distill them into a lower-dimensional latent space, after which the adaptive expansion operation is used to spatially align the data.
• Figure 5: Visualization results of LiteFusion. The ground truth and predictions on the BEV plane are drawn in green and blue rectangles, respectively. LiteFusion obtains a more accurate detection result than the original BEVFormer with minimal adjustment.