LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving

TL;DR

LIX tackles the problem of transferring spatial geometric priors learned by a RGB-X data-fusion teacher to a RGB-only student in autonomous driving semantic segmentation. It introduces two KD streams: dynamically-weighted logit distillation ($ abla$DWLD) that assigns per-logit weights to TCLD and NCLD, and adaptively-recalibrated feature distillation ($ abla$ARFD) that uses kernel-regression-based feature recalibration and centered kernel alignment (CKA/HSIC) to measure cross-network feature consistency. Key contributions include reformulating DKD into a per-logit weighted framework, a dynamic weight controller (DWC), Laplace-kernel feature recalibration, and HSIC-based CKA losses, all integrated in a joint loss $ \\mathcal{L} = \\mathcal{L}_{H} + \\lambda_{L} \\mathcal{L}_{L} + \\lambda_{F} \\mathcal{L}_{F} $. Extensive experiments on vKITTI2, KITTI Semantics, and nuImage demonstrate state-of-the-art performance and strong transfer of geometric priors from RGB-X teachers to RGB-only students, with practical implications for robust autonomous driving perception in sensor-constrained scenarios.

Abstract

Despite the impressive performance achieved by data-fusion networks with duplex encoders for visual semantic segmentation, they become ineffective when spatial geometric data are not available. Implicitly infusing the spatial geometric prior knowledge acquired by a data-fusion teacher network into a single-modal student network is a practical, albeit less explored research avenue. This article delves into this topic and resorts to knowledge distillation approaches to address this problem. We introduce the Learning to Infuse ''X'' (LIX) framework, with novel contributions in both logit distillation and feature distillation aspects. We present a mathematical proof that underscores the limitation of using a single, fixed weight in decoupled knowledge distillation and introduce a logit-wise dynamic weight controller as a solution to this issue. Furthermore, we develop an adaptively-recalibrated feature distillation algorithm, including two novel techniques: feature recalibration via kernel regression and in-depth feature consistency quantification via centered kernel alignment. Extensive experiments conducted with intermediate-fusion and late-fusion networks across various public datasets provide both quantitative and qualitative evaluations, demonstrating the superior performance of our LIX framework when compared to other state-of-the-art approaches.

Figures (7)

• Figure 1: An illustration of our proposed LIX framework, which consists of two key components: a) dynamically-weighted logit distillation and b) adaptively-recalibrated feature distillation.
• Figure 2: Qualitative comparison with other SoTA KD approaches on the KITTI Semantics dataset menze2015kitti using SNR-RoadSeg, where significantly improved areas are highlighted with white dashed boxes.
• Figure 3: Qualitative comparison with other SoTA KD approaches on the nuImage dataset nuscenes2019 using SNR-RoadSeg, where significantly improved areas are highlighted with white dashed boxes.
• Figure 4: Ablation studies on LD: (a) comparison between DKD zhao2022decoupled and DWLD with respect to different $\beta$, where "vK" and "KS" are the abbreviations of "vKITTI2" and "KITTI Semantics", respectively; (b) comparison among various designs of $\boldsymbol{\Omega}^{\mathcal{S}}$ on the vKITTI2 dataset.
• Figure 5: Comparison of the CKA scores between the features of baseline and distilled student networks.