Applications of Knowledge Distillation in Remote Sensing: A Survey

TL;DR

This review article provides an extensive examination of KD and its innovative applications in RS, including a comprehensive taxonomy of KD techniques, where each category is critically analyzed to demonstrate the breadth and depth of the alternative options.

Abstract

With the ever-growing complexity of models in the field of remote sensing (RS), there is an increasing demand for solutions that balance model accuracy with computational efficiency. Knowledge distillation (KD) has emerged as a powerful tool to meet this need, enabling the transfer of knowledge from large, complex models to smaller, more efficient ones without significant loss in performance. This review article provides an extensive examination of KD and its innovative applications in RS. KD, a technique developed to transfer knowledge from a complex, often cumbersome model (teacher) to a more compact and efficient model (student), has seen significant evolution and application across various domains. Initially, we introduce the fundamental concepts and historical progression of KD methods. The advantages of employing KD are highlighted, particularly in terms of model compression, enhanced computational efficiency, and improved performance, which are pivotal for practical deployments in RS scenarios. The article provides a comprehensive taxonomy of KD techniques, where each category is critically analyzed to demonstrate the breadth and depth of the alternative options, and illustrates specific case studies that showcase the practical implementation of KD methods in RS tasks, such as instance segmentation and object detection. Further, the review discusses the challenges and limitations of KD in RS, including practical constraints and prospective future directions, providing a comprehensive overview for researchers and practitioners in the field of RS. Through this organization, the paper not only elucidates the current state of research in KD but also sets the stage for future research opportunities, thereby contributing significantly to both academic research and real-world applications.

Figures (11)

• Figure 1: Summary of literature screening approach used in this review.
• Figure 2: An overview of the knowledge distillation principle.
• Figure 3: Principal steps of applying KD in RS applications.
• Figure 4: The YOLOv8n DT network architecture is structured into three primary components: the teacher network, the student network, and the distillation loss function module. This architecture incorporates both feature loss and logit loss within the distillation process to effectively transfer knowledge from the teacher to the student network, thereby enhancing the student's performance while maintaining efficiency.
• Figure 5: A Comprehensive Taxonomy of Existing KD Techniques.