Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantification of Large Language Model Distillation

Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xinrun Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, Shiwen Ni

TL;DR

The paper addresses the problem of quantifying large language model distillation and its tendency to homogenize model behavior. It introduces two metrics, Identity Consistency Evaluation (ICE) and Response Similarity Evaluation (RSE), to quantify identity leakage and cross-model response similarity, respectively, and pairs them with a jailbreak-based framework (GPTFuzz) to reveal distillation-driven vulnerabilities. Empirical results show that many well-known LLMs exhibit high distillation levels, with base models typically distilling more than aligned ones, while exceptions like Claude, Doubao, and Gemini demonstrate lower levels. The study emphasizes transparency in training and distillation data, highlighting implications for robustness and safety, and proposes a systematic approach for evaluating distillation that can inform independent development and policy. Overall, ICE and RSE provide a practical, interpretable framework to measure distillation effects, guiding future improvements in model transparency and diversity.

Abstract

Model distillation is a fundamental technique in building large language models (LLMs), transferring knowledge from a teacher model to a student model. However, distillation can lead to model homogenization, reducing diversity among models and impairing their ability to robustly handle complex or novel tasks. These limitations underscore the need to systematically quantify the distillation process and its impact. In this work, we propose a framework to evaluate and quantify model distillation. Our method addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information, and (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. Experimental results demonstrate two key insights: (1) Well-known closed-source and open-source LLMs usually exhibit high distillation degrees, except for Claude, Doubao, and Gemini. (2) Base LLMs show higher distillation degrees compared to aligned LLMs. By offering a systematic approach to improve the transparency of LLM data distillation, we call for LLMs with more independent development and more transparent technical reports to improve LLMs' robustness and safety. The code and data are available under https://github.com/Aegis1863/LLMs-Distillation-Quantification.

Quantification of Large Language Model Distillation

TL;DR

The paper addresses the problem of quantifying large language model distillation and its tendency to homogenize model behavior. It introduces two metrics, Identity Consistency Evaluation (ICE) and Response Similarity Evaluation (RSE), to quantify identity leakage and cross-model response similarity, respectively, and pairs them with a jailbreak-based framework (GPTFuzz) to reveal distillation-driven vulnerabilities. Empirical results show that many well-known LLMs exhibit high distillation levels, with base models typically distilling more than aligned ones, while exceptions like Claude, Doubao, and Gemini demonstrate lower levels. The study emphasizes transparency in training and distillation data, highlighting implications for robustness and safety, and proposes a systematic approach for evaluating distillation that can inform independent development and policy. Overall, ICE and RSE provide a practical, interpretable framework to measure distillation effects, guiding future improvements in model transparency and diversity.

Abstract

Model distillation is a fundamental technique in building large language models (LLMs), transferring knowledge from a teacher model to a student model. However, distillation can lead to model homogenization, reducing diversity among models and impairing their ability to robustly handle complex or novel tasks. These limitations underscore the need to systematically quantify the distillation process and its impact. In this work, we propose a framework to evaluate and quantify model distillation. Our method addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information, and (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. Experimental results demonstrate two key insights: (1) Well-known closed-source and open-source LLMs usually exhibit high distillation degrees, except for Claude, Doubao, and Gemini. (2) Base LLMs show higher distillation degrees compared to aligned LLMs. By offering a systematic approach to improve the transparency of LLM data distillation, we call for LLMs with more independent development and more transparent technical reports to improve LLMs' robustness and safety. The code and data are available under https://github.com/Aegis1863/LLMs-Distillation-Quantification.
Paper Structure (34 sections, 1 equation, 21 figures, 10 tables)

This paper contains 34 sections, 1 equation, 21 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Method
  4. Identity Consistency Evaluation
  5. Response Similarity Evaluation
  6. Experiment
  7. Experimental Settings
  8. Identity Consistency Evaluation
  9. Response Similarity Evaluation
  10. Experimental Results
  11. Main result of ICE
  12. Main result of RSE
  13. Case Study
  14. Related Work
  15. Knowledge Distillation.
  16. ...and 19 more sections

Figures (21)

  • Figure 1: Evaluation results of our method. (a) A Case of ICE. (b) Quantitative comparison of RSE and ICE, the distillation level increases towards the top-right corner, with Claude, Doubao, and Gemini showing low levels of distillation.
  • Figure 2: The distillation quantification framework, consisting of two main components: (1) ICE on the left, which assesses the misconception of identity-related traits in distilled models, and (2) RSE on the right, which measures the similarity of responses between target and reference models.
  • Figure 3: Scoring criteria of LLM-as-a-judge for RSE. This figure illustrates the five scoring levels used in RSE, ranging from 1 (very dissimilar) to 5 (very similar).
  • Figure 4: Identity Consistency Evaluation comparison. The mapping of the model abbreviations is as follows: 'Claude' corresponds to 'Claude3.5-Sonnet', 'Doubao' corresponds to 'Doubao-Pro-32k', 'Gemini' corresponds to 'Gemini-2.0-Flash', 'Llama3.1' corresponds to 'Llama3.1-70B-Instruct', 'DPSK-V3' corresponds to 'DeepSeek-V3', and 'Qwen-Max' corresponds to 'Qwen-Max-0919'.
  • Figure 5: Number of Identity Consistency Evaluation due to different types of identity prompts. Model abbreviation mapping is the same as Figure \ref{['fig:jailbreak_main_result']}.
  • ...and 16 more figures