Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization

TL;DR

This work tackles the challenge of transferring the strong zero-/few-shot generalization of vision-language models to task-specific learners under semi-supervised data. It introduces Dual-Head Optimization (DHO), which decouples supervised and distillation signals via two separate heads, mitigating gradient conflicts and yielding more robust feature representations. Through extensive experiments on 15 datasets, DHO consistently outperforms conventional KD baselines and achieves state-of-the-art results on ImageNet under low-shot settings and on OOD benchmarks when paired with existing adaptation methods, all with minimal computational overhead. The approach supports flexible inference-time mixing of head outputs and language-aware initialization, making it practical and readily integrable with existing VLM adaptation pipelines. The authors also release code and checkpoints to facilitate reproducibility and future research.

Abstract

Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Recently, vision-language models (VLMs), pre-trained on massive image-text pairs, have demonstrated remarkable zero-/few-shot performance that often surpasses SSL approaches due to their exceptional generalization capabilities. This gap motivates us to question: how can we effectively harness the powerful generalization capabilities of VLMs into task-specific models? Knowledge distillation (KD) offers a natural framework for transferring VLM capabilities, but we identify that it suffers from gradient conflicts between supervised and distillation losses. To address this challenge, we propose Dual-Head Optimization (DHO), which introduces dual prediction heads for each distinct signal. We observe that DHO resolves gradient conflicts, enabling improved feature learning compared to single-head KD baselines, with practical benefits of minimal computational overhead and test-time hyperparameter tuning without retraining. Extensive experiments across 15 datasets show that DHO consistently outperforms KD baselines, often outperforming teacher models with smaller student models. DHO also achieves new state-of-the-art performance on both in-distribution ImageNet semi-supervised learning and out-of-distribution generalization across ImageNet variants. We publicly release our code and model checkpoints to facilitate future research at https://github.com/erjui/DHO.

Figures (18)

• Figure 1: (Left): DHO consistently outperforms single-head baselines on 11 datasets under 16-shot semi-supervised setting. The improvements are evaluated in comparison to the second-best one. (Right): DHO achieves new SoTA on ImageNet in both 1% and 10% labeled data setting, with fewer parameters.
• Figure 2: Conceptual illustration on KD frameworks, Single-Head Optimization (SHO) and Dual-Head Optimization (DHO), for semi-supervised settings. As demonstrated in \ref{['fig:average_gradient_conflict']}, we observe gradient conflict of SHO. In contrast, DHOmitigates such conflicts by leveraging dual-head architectures in \ref{['fig:average_gradient_conflict']}.
• Figure 3: The average cosine similarity and inner product over 10 datasets.
• Figure 4: Results on 10 datasets under few-shot semi-supervision using ResNet-18 with zero-shot teacher.
• Figure 5: Results on 10 datasets using ResNet-18 with either zero- or few-shot teacher.

Theorems & Definitions (8)

• Theorem 1: Inference equivalence
• Theorem 2: Optimal Distribution for Single-Head Optimization
• proof
• Theorem 3: Inference Equivalence Under $\varepsilon$-Convergence
• proof
• Lemma 1: Temperature Matching via KL Divergence
• proof
• Corollary 1: Optimal DHO Configuration