Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation

TL;DR

The paper tackles the high computational load of BEV-based multimodal perception for autonomous driving by introducing a content-aware, joint input pruning method that uses the BEV representation as a shared anchor to prune non-critical regions across camera and LiDAR inputs. A pruning index predictor, trained end-to-end with the perception model, outputs a binary mask applied via an index multiplier to prune raw sensor data before sparse backbones, reducing complexity while preserving performance. Evaluations on NuScenes demonstrate that pruning up to 50% of inputs yields about a 35% reduction in GFlops and a 31% latency decrease with only small drops in 3D detection and map segmentation accuracy. The method outperforms traditional pruning baselines and demonstrates the practicality of input-level pruning for BEV-based perception pipelines, paving the way for more efficient on-board autonomous systems.

Abstract

In the landscape of autonomous driving, Bird's-Eye-View (BEV) representation has recently garnered substantial academic attention, serving as a transformative framework for the fusion of multi-modal sensor inputs. This BEV paradigm effectively shifts the sensor fusion challenge from a rule-based methodology to a data-centric approach, thereby facilitating more nuanced feature extraction from an array of heterogeneous sensors. Notwithstanding its evident merits, the computational overhead associated with BEV-based techniques often mandates high-capacity hardware infrastructures, thus posing challenges for practical, real-world implementations. To mitigate this limitation, we introduce a novel content-aware multi-modal joint input pruning technique. Our method leverages BEV as a shared anchor to algorithmically identify and eliminate non-essential sensor regions prior to their introduction into the perception model's backbone. We validatethe efficacy of our approach through extensive experiments on the NuScenes dataset, demonstrating substantial computational efficiency without sacrificing perception accuracy. To the best of our knowledge, this work represents the first attempt to alleviate the computational burden from the input pruning point.

Figures (5)

• Figure 1: Illustration of Multi-Modal Projection Relationship. The areas to be removed in the sensor space can be inferred from the anchor position in the BEV space via the intrinsic and extrinsic parameters of the sensors
• Figure 2: Overall Training Architecture. This system combines a multi-modal perception model and a pruning index predictor. During the training phase, the predictor is trained along with the perception model in an end-to-end manner. The size of the input voxel grids, pruning index and BEV feature map are set to be identical to serve the purpose of joint pruning of multi-modal inputs.
• Figure 3: Overall Inference Architecture. In the testing phase, the predictor initially generates a pruning mask, as depicted in module (a). Subsequently, it employs the index multiplier, illustrated in module (c), to align the pruning mask with the voxelized input, shown in module (b), before proceeding to the sparse backbone in module (d).
• Figure 4: Comparison of Performance Change w.r.t. Cropping Ratio. Models are trained for 12 epochs instead of 24 for fast verification purposes.
• Figure 5: Comparison of Performance Change w.r.t. Number of images used to generate pruning mask. Pruning ratio is set to be 50% for fair comparison. Models are trained for 12 epochs instead of 24 for fast verification purpose.