Weak-to-Strong 3D Object Detection with X-Ray Distillation

TL;DR

The paper tackles sparsity and occlusion in LiDAR-based 3D object detection by introducing the X-Ray Teacher framework, which leverages Object-Complete Frames generated from temporal LiDAR sequences. A Teacher trained on these complete frames distills knowledge to a stronger Student operating on original data, enabling universal applicability across detectors. In supervised settings, the approach uses KD losses on heads, embeddings, and detections, while in semi-supervised settings, Objects Temporal Fusion generates Object-Complete frames for unlabeled data and enables pseudo-label-based distillation without EMA. Across NuScenes, Waymo, and ONCE, the method yields 1–2 mAP gains in supervised models and 0.8–1.4 mAP improvements on ONCE, achieving state-of-the-art performance on semi-supervised benchmarks and demonstrating broad effectiveness for sequential LiDAR-based detection.

Abstract

This paper addresses the critical challenges of sparsity and occlusion in LiDAR-based 3D object detection. Current methods often rely on supplementary modules or specific architectural designs, potentially limiting their applicability to new and evolving architectures. To our knowledge, we are the first to propose a versatile technique that seamlessly integrates into any existing framework for 3D Object Detection, marking the first instance of Weak-to-Strong generalization in 3D computer vision. We introduce a novel framework, X-Ray Distillation with Object-Complete Frames, suitable for both supervised and semi-supervised settings, that leverages the temporal aspect of point cloud sequences. This method extracts crucial information from both previous and subsequent LiDAR frames, creating Object-Complete frames that represent objects from multiple viewpoints, thus addressing occlusion and sparsity. Given the limitation of not being able to generate Object-Complete frames during online inference, we utilize Knowledge Distillation within a Teacher-Student framework. This technique encourages the strong Student model to emulate the behavior of the weaker Teacher, which processes simple and informative Object-Complete frames, effectively offering a comprehensive view of objects as if seen through X-ray vision. Our proposed methods surpass state-of-the-art in semi-supervised learning by 1-1.5 mAP and enhance the performance of five established supervised models by 1-2 mAP on standard autonomous driving datasets, even with default hyperparameters. Code for Object-Complete frames is available here: https://github.com/sakharok13/X-Ray-Teacher-Patching-Tools.

Figures (9)

• Figure 1: 3D object detection directly from sparse LiDAR data (top left) provides noisy predictions (bottom left). Adding object completion stage (top right) helps to train 3D object detection X- Ray Teacher, which is robust and can be distilled to baseline model. Red, Yellow and Blue colors of bounding boxes are related to classical LiDAR-based object detection, Our model on Object-Complete frames predictions and Ground Truth labels, respectively.
• Figure 2: Overall X-Ray Knowledge Distillation for Supervised Learning. X-Ray Teacher is frozen and pretrained on Object-Complete frames which are taken as input. The Student is guided to mimic the Teacher's behaviour through Knowledge Distillation losses: $L_{feat}$ for intermediate embeddings matching, $L_{reg}$ for bounding box regression, $L_{det}$ for basic detection, and $L_{cls}$ for classification.
• Figure 3: Object-Complete Frame generation process for semi-supervised setting. It consists of tracking and Point Cloud Registration. We track objects across all frames in the whole sequence, then we use Point Cloud Registration to merge points that represent the same object from different views, and finally we replace the original object with the new, complete one.
• Figure 4: Semi-supervised X-Ray Teacher pipeline for 3D object detection task. Unlabeled sequences are processed by the Objects Temporal Fusion block to create more complete object representations by aggregating information over time. The Student model learns from both pseudo-labeled predictions and actual labeled data with ground truth annotations.
• Figure 5: Visual comparison between noisy and poor baseline (original) 3D detector SECOND (left column) and our X-Ray Teacher that perceives Object-Complete frames. We compare two identical timestamps and view angles. The Baseline model fails to detect some objects while X-Ray Teacher does not. This explains why knowledge distillation is indeed beneficial and should improve models.