SAMKD: Spatial-aware Adaptive Masking Knowledge Distillation for Object Detection

TL;DR

SAMKD tackles the challenge of knowledge distillation for object detection by moving beyond global feature masking to a spatially aware, coarse-to-fine distillation framework. It combines a spatial pyramid-based spatially hierarchical feature masking distillation (SFD) with a region-aware, adaptive logit distillation (SLD), guided by dual teacher attention maps and feature discrepancies. The method yields consistent improvements over strong baselines across multiple detectors on COCO, with notable gains for small objects and reduced teacher-student gaps. These contributions offer a practical, scalable approach to compressing powerful detectors while preserving fine-grained localization cues.

Abstract

Most of recent attention-guided feature masking distillation methods perform knowledge transfer via global teacher attention maps without delving into fine-grained clues. Instead, performing distillation at finer granularity is conducive to uncovering local details supplementary to global knowledge transfer and reconstructing comprehensive student features. In this study, we propose a Spatial-aware Adaptive Masking Knowledge Distillation (SAMKD) framework for accurate object detection. Different from previous feature distillation methods which mainly perform single-scale feature masking, we develop spatially hierarchical feature masking distillation scheme, such that the object-aware locality is encoded during coarse-to-fine distillation process for improved feature reconstruction. In addition, our spatial-aware feature distillation strategy is combined with a masking logit distillation scheme in which region-specific feature difference between teacher and student networks is utilized to adaptively guide the distillation process. Thus, it can help the student model to better learn from the teacher counterpart with improved knowledge transfer and reduced gap. Extensive experiments for detection task demonstrate the superiority of our method. For example, when FCOS is used as teacher detector with ResNet101 backbone, our method improves the student network from 35.3\% to 38.8\% mAP, outperforming state-of-the-art distillation methods including MGD, FreeKD and DMKD.

Figures (4)

• Figure 1: Illustration of feature maps obtained by our proposed SAMKD. Benefiting from the spatially hierarchical distillation mechanism, our model can encode region-specific semantics at different feature granularities, significantly improving object awareness and facilitating model adaptability. Specifically, when detecting multiple horses in the image, our model can focus on their bodies and legs at finer $4\times4$ scale, leading to the feature heatmap with high responses to the central regions.
• Figure 2: Illustration of our proposed SAMKD framework. For both the teacher and the student models, spatial pyramid mechanism is integrated into feature pyramid networks (FPN) from scratch and a spatial-aware feature masking distillation scheme is developed. Specifically, the masking regions on the student feature maps are obtained by dual attention maps generated from the teacher model at each level and comprehensive student feature reconstruction can be achieved via hierarchical feature distillation. Furthermore, the differences between the teacher and the student feature maps within respective divided regions are adopted as auxiliary masking weights for facilitating coarse-to-fine logit distillation, leading to accurate predictions of the student logits.
• Figure 3: Visualizations of different feature maps. Given the original image (a), our coarse-to-fine distillation method (e) can capture more critical object-aware clues (e.g., bus roof and side) compared with performing distillation on individual scale $1\times1$ (b), $2\times2$ (c), and $4\times4$ (d), and thus achieves more accurate detection results. In implementation, the teacher and the student models are GFL-ResNeXt101 and GFL-ResNet50.
• Figure 4: Performance of our SAMKD using different values of $\alpha$ and $\beta$ with GFL-ResNeXt101 (teacher) and GFL-ResNet50 (student).