TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation

TL;DR

TIE-KD addresses the high computational cost of state-of-the-art monocular depth estimation by introducing a teacher-independent, explainable knowledge distillation framework. It constructs a Depth Probability Map (DPM) from the teacher's depth output and distills knowledge to a flexible student using two losses: a DPM-based KL divergence and a depth-map SSIM loss, enabling effective transfer without architectural constraints. Across KITTI experiments with three diverse teachers, TIE-KD consistently outperforms traditional response-based KD, and demonstrates robustness across backbones and varying hyperparameters, while preserving interpretability via the DPM. This approach promises more practical, efficient, and transparent depth estimation deployments in real-world systems.

Abstract

Monocular depth estimation (MDE) is essential for numerous applications yet is impeded by the substantial computational demands of accurate deep learning models. To mitigate this, we introduce a novel Teacher-Independent Explainable Knowledge Distillation (TIE-KD) framework that streamlines the knowledge transfer from complex teacher models to compact student networks, eliminating the need for architectural similarity. The cornerstone of TIE-KD is the Depth Probability Map (DPM), an explainable feature map that interprets the teacher's output, enabling feature-based knowledge distillation solely from the teacher's response. This approach allows for efficient student learning, leveraging the strengths of feature-based distillation. Extensive evaluation of the KITTI dataset indicates that TIE-KD not only outperforms conventional response-based KD methods but also demonstrates consistent efficacy across diverse teacher and student architectures. The robustness and adaptability of TIE-KD underscore its potential for applications requiring efficient and interpretable models, affirming its practicality for real-world deployment.

Figures (8)

• Figure 1: Comparative visualization of depth estimation results showcasing the effectiveness of the proposed TIE-KD framework. The first row displays input images; the second row depicts outcomes from a baseline small model trained on ground truth. The third row shows results from the high-capacity teacher model, AdaBins bhat2021adabins. The fourth and fifth rows illustrate depth maps from students trained via a response-based KD and our TIE-KD, respectively. Our TIE-KD demonstrates a more effective knowledge distillation performance than prior response-based KD methods, achieving greater similarity to the teacher model, especially in preserving edge definition and depth accuracy.
• Figure 2: Overview of our teacher-independent and explainable KD process for single image depth estimation
• Figure 3: Visual comparison of depth maps across various models for three different scenes, highlighting detailed variances within the regions enclosed by green boxes.
• Figure 4: Impact of loss function weight ($\alpha$) on TIE-KD performance, with each subfigure representing a different metric.
• Figure 5: Performance variation of the TIE-KD framework with respect to the standard deviation ($\sigma$) used in depth probability map generation. Each subfigure presents a different metric.