MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection

TL;DR

MonoTAKD tackles depth ambiguity in monocular 3D object detection by introducing a camera-based Teaching Assistant (TA) that provides robust 3D visual knowledge to a camera-based student via end-to-end knowledge distillation. The framework combines intra-modal distillation (IMD) to transfer visual 3D cues and cross-modal residual distillation (CMRD) to convey LiDAR-exclusive spatial cues, aided by a Spatial Alignment Module (SAM) and a Feature Fusion Module (FFM) to refine BEV representations. Training optimizes a composite loss $L_{total} = L_{IMD} + L_{CMRD} + L_{logit}$, enabling strong performance on KITTI3D and robust generalization to nuScenes and KITTI raw, while keeping training costs practical. Overall, MonoTAKD provides a principled, end-to-end distillation framework that narrows cross-modality gaps and advances practical monocular 3D perception for autonomous driving.

Abstract

Monocular 3D object detection (Mono3D) holds noteworthy promise for autonomous driving applications owing to the cost-effectiveness and rich visual context of monocular camera sensors. However, depth ambiguity poses a significant challenge, as it requires extracting precise 3D scene geometry from a single image, resulting in suboptimal performance when transferring knowledge from a LiDAR-based teacher model to a camera-based student model. To facilitate effective distillation, we introduce Monocular Teaching Assistant Knowledge Distillation (MonoTAKD), which proposes a camera-based teaching assistant (TA) model to transfer robust 3D visual knowledge to the student model, leveraging the smaller feature representation gap. Additionally, we define 3D spatial cues as residual features that capture the differences between the teacher and the TA models. We then leverage these cues to improve the student model's 3D perception capabilities. Experimental results show that our MonoTAKD achieves state-of-the-art performance on the KITTI3D dataset. Furthermore, we evaluate the performance on nuScenes and KITTI raw datasets to demonstrate the generalization of our model to multi-view 3D and unsupervised data settings. Our code is available at https://github.com/hoiliu-0801/MonoTAKD.

Figures (7)

• Figure 1: Comparison between previous methods and MonoTAKD (Ours). Previous works cmkdmonodis face a significant challenge in the distillation process due to a substantial gap in feature representation. Our MonoTAKD incorporates intra-modal and cross-modal residual distillation to enhance learning across this feature representation gap. We visualize the BEV features of each model, along with the residual features derived between the TA and T models, highlighted in orange. Best view with zoom-in and color.
• Figure 2: Overall architecture of the MonoTAKD . The top, middle, and bottom rows show the architecture of the LiDAR-based teacher, the camera-based teaching assistant (TA), and a camera-based student. We design the intra-modal distillation (IMD) and the cross-modal residual distillation (CMRD) processes to guide the camera-based student. In addition, a spatial alignment module (SAM) and a feature fusion module (FFM) are employed to improve the BEV feature representation.
• Figure 3: Spatial Alignment Module (SAM). SAM cascades the Atrous and Deformable convolutions to learn the alignment of BEV features. SENet is adopted for channel attention.
• Figure 4: Comparing the BEV features ($F^{T}$) from the teacher model with the residual features ($F_{res}$) reveals that our residuals effectively capture essential 3D spatial cues, emphasizing critical information over less important elements, such as ripples and background noise present in $F^{T}$.
• Figure 5: Convergence curves of different feature distillation on the KITTI val set. The x-axis shows the number of epochs, and the y-axis denotes $AP_{3D}$ for the Car category at the moderate level.