Domain-invariant Progressive Knowledge Distillation for UAV-based Object Detection

TL;DR

This work tackles UAV-based object detection under tight compute constraints by addressing two core issues in knowledge distillation: the large scale gap between teacher and student models and domain variations from complex UAV scenes. It introduces a progressive distillation framework with a senior and a junior teacher to gradually transfer knowledge, coupled with a domain-invariant transfer using Fast Fourier Transform to separate amplitude (domain-dependent) and phase (domain-invariant) information. The proposed approach achieves state-of-the-art results on VisDrone and SynDrone, outperforming prior KD methods and even surpassing the teacher in some metrics, while ablations highlight the additive gains from both components. The findings suggest that combining progressive teaching with FFT-based domain-invariant transfer can robustly improve lightweight detectors for UAV-OD in diverse environments.

Abstract

Knowledge distillation (KD) is an effective method for compressing models in object detection tasks. Due to limited computational capability, UAV-based object detection (UAV-OD) widely adopt the KD technique to obtain lightweight detectors. Existing methods often overlook the significant differences in feature space caused by the large gap in scale between the teacher and student models. This limitation hampers the efficiency of knowledge transfer during the distillation process. Furthermore, the complex backgrounds in UAV images make it challenging for the student model to efficiently learn the object features. In this paper, we propose a novel knowledge distillation framework for UAV-OD. Specifically, a progressive distillation approach is designed to alleviate the feature gap between teacher and student models. Then a new feature alignment method is provided to extract object-related features for enhancing student model's knowledge reception efficiency. Finally, extensive experiments are conducted to validate the effectiveness of our proposed approach. The results demonstrate that our proposed method achieves state-of-the-art (SoTA) performance in two UAV-OD datasets.

Figures (3)

• Figure 1: Motivations of our proposed method. (a) The complex environments of UAVs lead to significant domain variations. For instance, factors such as background, lighting, perspective, and weather conditions contribute to the difficulty of UAV-OD. (b) We conducted CKA analysis on features of models at different scales (YOLOv7-X, L, Tiny). The results indicate that closer-scale model features have smaller differences, which are more conducive to knowledge transfer.
• Figure 2: Overview of our proposed method. We leverage two novel distillation methods for the UAV-OD task. Step 1: We introduce two different scale teacher models (senior teacher and junior teacher) and adopt a progressive distillation strategy. Step 2: We propose a domain-invariant knowledge transferring approach by utilizing fast fourier transform. Our approach can effectively excavate the potential of the student model in UAV-OD, enabling it to reach or exceed the junior teacher model's accuracy.
• Figure 3: mAP curves during distillation on SynDrone and VisDrone Datasets.