Distilling Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection

TL;DR

This work tackles the efficiency-accuracy tension in camera-based BEV 3D object detection by introducing TempDistiller, a temporal knowledge distillation framework that transfers long-term temporal memory from a strong teacher to a lightweight student using masked temporal feature reconstruction on sparse BEV representations. The method also leverages temporal relational distillation when full-frame inputs are used, enabling robust velocity estimation without additional inference cost. Extensive experiments on nuScenes demonstrate consistent performance gains (e.g., higher mAP and NDS) and a notable speedup when reducing input frames, with qualitative results showing improved detection of occluded and distant objects. Overall, TempDistiller provides a practical approach to imbue lightweight detectors with long-term temporal knowledge, improving accuracy and efficiency in camera-only 3D object detection.

Abstract

Striking a balance between precision and efficiency presents a prominent challenge in the bird's-eye-view (BEV) 3D object detection. Although previous camera-based BEV methods achieved remarkable performance by incorporating long-term temporal information, most of them still face the problem of low efficiency. One potential solution is knowledge distillation. Existing distillation methods only focus on reconstructing spatial features, while overlooking temporal knowledge. To this end, we propose TempDistiller, a Temporal knowledge Distiller, to acquire long-term memory from a teacher detector when provided with a limited number of frames. Specifically, a reconstruction target is formulated by integrating long-term temporal knowledge through self-attention operation applied to feature teachers. Subsequently, novel features are generated for masked student features via a generator. Ultimately, we utilize this reconstruction target to reconstruct the student features. In addition, we also explore temporal relational knowledge when inputting full frames for the student model. We verify the effectiveness of the proposed method on the nuScenes benchmark. The experimental results show our method obtain an enhancement of +1.6 mAP and +1.1 NDS compared to the baseline, a speed improvement of approximately 6 FPS after compressing temporal knowledge, and the most accurate velocity estimation.

Figures (4)

• Figure 1: Overview of masked feature reconstruction. (a) Random masks are generated on student features. Then spatial features are recovered from the teacher. (b) Instance masks are introduced to filter foreground areas. Random masks are then generated within these specific areas and student features are reconstructed from the teacher. (c) The proposed temporal feature reconstruction is based on sparse BEV representation. The reconstruction target involves temporal aggregated features derived from a teacher model, facilitating the acquisition of long-term temporal knowledge by student features.
• Figure 2: Overall framework of TempDistiller. The proposed method aims to enable student detector to learn long-term temporal knowledge from teacher detector, particular with fewer input frames. Taking the long-term temporal teacher features as the reconstruction objective, we leverage temporal feature reconstruction (TFR) to force student detector to study enhanced representation of perspective features and BEV features. Additionally, we explore temporal relational distillation (TRD) when the student detector is fed with full frames. Finally, we impose constraints on the features after S&T (Spatio-Temporal) decoder by MSE (mean square error), which encourages the learning of semantically rich spatio-temporal features. SSA refers to the spatial self-attention operation in liu2023sparsebev.
• Figure 3: Overview of the proposed temporal relational distillation (TRD). $\odot$ indicates element-wise product. Objects that are related in different frames have high responses in the similarity matrix.
• Figure 4: Qualitative results over three consecutive frames (front camera) in two scenes. The first and third row show the prediction made by the baseline model, while the second and fourth row demonstrate the predictive results of TempDistiller. In the last column, the LiDAR point cloud in BEV is display for frame $t+1$, except the last row (for $t+2$) due to the limited BEV distance. TempDistiller successfully predicts an occluded car merging into the main road and a pedestrian crossing the street in the distance, highlighted by red dotted circles.