Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning

Rongrong Xie, Yizhou Xu, Guido Sanguinetti

TL;DR

The paper tackles when cross-modal knowledge distillation (KD) yields gains by introducing the Cross-modal Complementarity Hypothesis (CCH): cross-modal KD is effective if the mutual information between teacher and student representations, $I(H_1;H_2)$, exceeds the mutual information between the student representations and the labels, $I(H_2;Y)$ ($I(H_1;H_2) > I(H_2;Y)$). It provides a theoretical validation in a joint Gaussian model and demonstrates empirical support across synthetic data, image, video-text-audio, and cancer omics modalities, indicating that MI gaps reliably predict KD efficacy. The work further offers practical guidelines for selecting teacher modalities and fusion strategies (e.g., Fusion+KD) based on MI relationships, and uses multiple MI estimators to robustly quantify information sharing. Overall, CCH links a concrete information-theoretic criterion to the success of cross-modal KD and delivers a principled approach to leveraging privileged modalities in multimodal learning with tangible benefits for real-world tasks.

Abstract

The rapid increase in multimodal data availability has sparked significant interest in cross-modal knowledge distillation (KD) techniques, where richer "teacher" modalities transfer information to weaker "student" modalities during model training to improve performance. However, despite successes across various applications, cross-modal KD does not always result in improved outcomes, primarily due to a limited theoretical understanding that could inform practice. To address this gap, we introduce the Cross-modal Complementarity Hypothesis (CCH): we propose that cross-modal KD is effective when the mutual information between teacher and student representations exceeds the mutual information between the student representation and the labels. We theoretically validate the CCH in a joint Gaussian model and further confirm it empirically across diverse multimodal datasets, including image, text, video, audio, and cancer-related omics data. Our study establishes a novel theoretical framework for understanding cross-modal KD and offers practical guidelines based on the CCH criterion to select optimal teacher modalities for improving the performance of weaker modalities.

Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning

TL;DR

The paper tackles when cross-modal knowledge distillation (KD) yields gains by introducing the Cross-modal Complementarity Hypothesis (CCH): cross-modal KD is effective if the mutual information between teacher and student representations, , exceeds the mutual information between the student representations and the labels, (). It provides a theoretical validation in a joint Gaussian model and demonstrates empirical support across synthetic data, image, video-text-audio, and cancer omics modalities, indicating that MI gaps reliably predict KD efficacy. The work further offers practical guidelines for selecting teacher modalities and fusion strategies (e.g., Fusion+KD) based on MI relationships, and uses multiple MI estimators to robustly quantify information sharing. Overall, CCH links a concrete information-theoretic criterion to the success of cross-modal KD and delivers a principled approach to leveraging privileged modalities in multimodal learning with tangible benefits for real-world tasks.

Abstract

The rapid increase in multimodal data availability has sparked significant interest in cross-modal knowledge distillation (KD) techniques, where richer "teacher" modalities transfer information to weaker "student" modalities during model training to improve performance. However, despite successes across various applications, cross-modal KD does not always result in improved outcomes, primarily due to a limited theoretical understanding that could inform practice. To address this gap, we introduce the Cross-modal Complementarity Hypothesis (CCH): we propose that cross-modal KD is effective when the mutual information between teacher and student representations exceeds the mutual information between the student representation and the labels. We theoretically validate the CCH in a joint Gaussian model and further confirm it empirically across diverse multimodal datasets, including image, text, video, audio, and cancer-related omics data. Our study establishes a novel theoretical framework for understanding cross-modal KD and offers practical guidelines based on the CCH criterion to select optimal teacher modalities for improving the performance of weaker modalities.
Paper Structure (19 sections, 3 theorems, 37 equations, 5 figures, 21 tables, 2 algorithms)

This paper contains 19 sections, 3 theorems, 37 equations, 5 figures, 21 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Unimodal KD
  4. Cross‐Modal KD
  5. Modality gaps
  6. Theoretical foundations
  7. The Cross-modal Complementarity Hypothesis
  8. Experiments
  9. Synthetic data
  10. Image data
  11. CMU-MOSEI dataset
  12. Cancer data
  13. Conclusion
  14. Theoretical analysis
  15. Experimental details and results for synthetic data
  16. ...and 4 more sections

Key Result

Theorem 1

Assume that $\kappa>1$ and $w_1^T\Sigma_{11}w_1\leq\Sigma_{33}$, $w_1^T\Sigma_{13}\geq0$. Suppose that $I(w_1^Tx_1,(w^*)^Tx_2)>I((w^*)^Tx_2,y)$, where $w^*:=\Sigma_{22}^{-1}\Sigma_{23}$ is the optimal student weight, then we have asymptotically for small $\lambda$.

Figures (5)

  • Figure 1: Synthetic regression experiments. When $I(H_1;H_2)$ exceeds $I(H_2;Y)$, the KD-trained student achieves lower test MSE than a non-distilled student; otherwise, KD provides no improvement.
  • Figure 2: Relationship between student accuracy and mutual information under varying Gaussian blur. (a) Test accuracy of the MNIST--M student trained with (solid line) and without (dashed line) distillation as a function of Gaussian blur standard deviation $\gamma$ applied to MNIST teacher inputs. (b) Mutual information $I_{TS}=I(H_{\mathrm{MNIST}};H_{\mathrm{MNIST\text{-}M}})$ (red) and $I_{SL}=I(H_{\mathrm{MNIST\text{-}M}};Y)$ (purple) versus $\gamma$. Accuracy improvements align with the region where $I_{TS}>I_{SL}$. For reference, $I_{TL}=I(H_{\mathrm{MNIST}};Y)=2.0485$, and the teacher network attains a test accuracy of 0.981.
  • Figure 3: Multimodal fusion architectures: direct fusion (left) and Fusion+KD (right).
  • Figure 4: Test MSE on synthetic regression data for varying distillation weight $\lambda$. Orange dashed curves: student with KD; blue dashed curves: student without KD.
  • Figure 5: Sample images from MNIST (left) and MNIST-M (right).

Theorems & Definitions (5)

  • Theorem 1
  • Theorem 2
  • proof
  • Theorem 3
  • proof