Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving

TL;DR

Drive-KD addresses the resource-intensive nature of large VLMs in autonomous driving by decomposing driving into perception, reasoning, and planning and transferring these capabilities via targeted attention-based distillation. It systematically selects distillation layers and signals, showing attention-based supervision to be more stable than hidden states and excluding output-distribution alignment due to driving outputs' diffuse confidence. The framework introduces three capability-specialized teachers and asymmetric gradient projection to mitigate cross-capability conflicts, achieving strong efficiency gains (e.g., $42\times$ memory reduction and $11.4\times$ throughput increase for InternVL3-1B) while outperforming larger pretrained models on DriveBench and GPT-5.1 on planning. Drive-KD generalizes across model families and scales, enabling practical, high-performance autonomous driving VLMs, though reasoning remains the most challenging capability to transfer to smaller models and warrants further study.

Abstract

Autonomous driving is an important and safety-critical task, and recent advances in LLMs/VLMs have opened new possibilities for reasoning and planning in this domain. However, large models demand substantial GPU memory and exhibit high inference latency, while conventional supervised fine-tuning (SFT) often struggles to bridge the capability gaps of small models. To address these limitations, we propose Drive-KD, a framework that decomposes autonomous driving into a "perception-reasoning-planning" triad and transfers these capabilities via knowledge distillation. We identify layer-specific attention as the distillation signal to construct capability-specific single-teacher models that outperform baselines. Moreover, we unify these single-teacher settings into a multi-teacher distillation framework and introduce asymmetric gradient projection to mitigate cross-capability gradient conflicts. Extensive evaluations validate the generalization of our method across diverse model families and scales. Experiments show that our distilled InternVL3-1B model, with ~42 times less GPU memory and ~11.4 times higher throughput, achieves better overall performance than the pretrained 78B model from the same family on DriveBench, and surpasses GPT-5.1 on the planning dimension, providing insights toward efficient autonomous driving VLMs.

Figures (14)

• Figure 1: Drive-KD multi-teacher distillation framework. Three teachers distill attention at different student layers: Perception/Planning uses first-layer/penultimate-layer cross-modal attention; Reasoning uses grouped-matching intermediate-layer full attention. The student is also trained with hard-label supervision at the output. We apply AGP to mitigate cross-capability gradient conflicts.
• Figure 2: Pre-study summary for InternVL3-8B: (a) layer-wise distillation alignment measured by cosine similarity (adjacent-layer and within-layer vision--text), (b) capability-wise intra-group similarity across layers, (c) layer-wise dispersion of hidden states and attention maps ($1-\cos$), and (d) position-normalized generalized margin along the answer segment comparing driving and general data at $\tau\!\approx\!1.0$.
• Figure 3: Asymmetric Gradient Projection (AGP). Stage 1 uses an asymmetric anchor--follower projection within each capability and merges the resulting update. Stage 2 applies shuffled symmetric pairwise projections across capabilities to obtain the final gradient direction (gradient B shown as an example).
• Figure 4: Layer-wise similarity profiles across model families and scales. Adjacent-layer cosine similarity (Adj. CosSim) and within-layer vision--text cosine similarity (V--T CosSim) are shown for InternVL3 (1B/8B/38B/78B) and Qwen2.5-VL (3B/7B/32B/72B).
• Figure 5: Capability-wise intra-consistency (mean pairwise cosine similarity) across Transformer layers for InternVL3 (top row) and Qwen2.5-VL (bottom row) under three task types: perception, reasoning, and planning.