Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning

TL;DR

Problem: road segmentation in very-high-resolution remote sensing with limited labeled data. Approach: a semi-supervised learning-based knowledge distillation framework (SSLKD) that uses two large teacher models to generate rich information and pseudo labels, guiding a lightweight student through backbone-feature alignment ($L_{dis}^{f}$), probability alignment ($L_{dis}^{p}$), and label-level distillation ($L_{unsup}$), with the total loss $L_{tot}=L_{sup}+L_{dis}^{f}+L_{dis}^{p}+L_{unsup}$. Key contributions: (i) a two-teacher cross-model supervision setup; (ii) integration of semi-supervised learning with knowledge distillation; (iii) empirical gains on RoadNet surpassing existing semi-supervised methods. Significance: demonstrates practical gains in accuracy and efficiency for road detection in remote sensing by leveraging unlabeled data.

Abstract

The advancement of knowledge distillation has played a crucial role in enabling the transfer of knowledge from larger teacher models to smaller and more efficient student models, and is particularly beneficial for online and resource-constrained applications. The effectiveness of the student model heavily relies on the quality of the distilled knowledge received from the teacher. Given the accessibility of unlabelled remote sensing data, semi-supervised learning has become a prevalent strategy for enhancing model performance. However, relying solely on semi-supervised learning with smaller models may be insufficient due to their limited capacity for feature extraction. This limitation restricts their ability to exploit training data. To address this issue, we propose an integrated approach that combines knowledge distillation and semi-supervised learning methods. This hybrid approach leverages the robust capabilities of large models to effectively utilise large unlabelled data whilst subsequently providing the small student model with rich and informative features for enhancement. The proposed semi-supervised learning-based knowledge distillation (SSLKD) approach demonstrates a notable improvement in the performance of the student model, in the application of road segmentation, surpassing the effectiveness of traditional semi-supervised learning methods.

Figures (4)

• Figure 1: Steps of the Knowledge Distillation Procedure.
• Figure 2: Framework of Cross-model Supervision.
• Figure 3: Framework of the proposed semi-supervised learning based knowledge distillation (SSLKD).
• Figure 4: Visual segmentation results of the student model in each method on the RoadNet dataset.