JointDistill: Adaptive Multi-Task Distillation for Joint Depth Estimation and Scene Segmentation

TL;DR

The paper tackles joint scene segmentation and depth estimation by introducing an adaptive multi-task distillation framework. It comprises two core components: feedback-based multi-teacher weighting, which dynamically adjusts knowledge transfer from task-specific teachers based on the student’s unseen validation performance, and trajectory-based distillation, which preserves essential past knowledge via top-K learning trajectories to mitigate forgetting. The student model consolidates a shared backbone with task-specific heads and relies on a connector to fuse teacher features, optimized with a composite loss including $L_S$ and $L_F$ alongside a trajectory term. Empirically, the approach improves performance on Cityscapes and NYU-v2 over strong baselines, with lower storage and computation costs due to the lightweight trajectory representation, and demonstrates potential for multimodal extensions.

Abstract

Depth estimation and scene segmentation are two important tasks in intelligent transportation systems. A joint modeling of these two tasks will reduce the requirement for both the storage and training efforts. This work explores how the multi-task distillation could be used to improve such unified modeling. While existing solutions transfer multiple teachers' knowledge in a static way, we propose a self-adaptive distillation method that can dynamically adjust the knowledge amount from each teacher according to the student's current learning ability. Furthermore, as multiple teachers exist, the student's gradient update direction in the distillation is more prone to be erroneous where knowledge forgetting may occur. To avoid this, we propose a knowledge trajectory to record the most essential information that a model has learnt in the past, based on which a trajectory-based distillation loss is designed to guide the student to follow the learning curve similarly in a cost-effective way. We evaluate our method on multiple benchmarking datasets including Cityscapes and NYU-v2. Compared to the state-of-the-art solutions, our method achieves a clearly improvement. The code is provided in the supplementary materials.

Figures (6)

• Figure 1: Compared to existing static distillation method, we adjust the amount of knowledge to be transferred from each single-task teacher in a dynamic manner based on the student's feedback. Such feedback reflects the true ability of the student on each task so that we could balance the multiple knowledge adaptively.
• Figure 2: Illustration of our proposed adaptive multi-task knowledge distillation method. The knowledge from multiple single-task teachers (two teachers in this application, one for segmentation and the other for depth estimation) is integrated and transferred to a multi-task student in a dynamic manner based on the periodical feedback on a separate validation dataset. Meanwhile a novel trajectory distillation is proposed to regulate the student to follow the most essential knowledge of multiple task-specific teachers with lower training cost.
• Figure 3: In our proposed trajectory distillation, we design a knowledge trajectory to capture the most essential information of the teacher model over the most recent $t$ time-periods, and require the student to follow the top-k trajectories to avoid knowledge forgetting problem.
• Figure 4: Visualization examples from Cityscapes for (Column 2-5) semantic segmentation and (Column 6-9) depth estimation.
• Figure 5: Ablation results of different $\alpha$ and different optimization learning rates $\beta$ (Eqn. \ref{['equ_omega']}) when updates $\omega$.