Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Symbolic Knowledge Distillation of Large Language Models

Kamal Acharya, Alvaro Velasquez, Houbing Herbert Song

TL;DR

This survey maps the growing field of symbolic knowledge distillation for large language models (LLMs), defining symbolic KD as translating implicit neural knowledge into explicit symbolic representations such as rules and graphs. It categorizes approaches into Direct, Multilevel, and RL-based distillation, and reviews how these methods interact with LLM architectures, training paradigms, and downstream applications. The paper synthesizes related works on knowledge bases, consistency, editing, reasoning, and explainability, highlighting advances in commonsense, translation, summarisation, mathematics, vision-language, and instruction generation. It also discusses opportunities, challenges, and practical implications, emphasizing data quality, human-machine collaboration, and the potential for open-source data and models. Overall, the survey argues that symbolic KD offers a viable, scalable path to more interpretable, efficient, and domain-tailored AI systems that complement ever-larger LLMs.

Abstract

This survey paper delves into the emerging and critical area of symbolic knowledge distillation in Large Language Models (LLMs). As LLMs like Generative Pre-trained Transformer-3 (GPT-3) and Bidirectional Encoder Representations from Transformers (BERT) continue to expand in scale and complexity, the challenge of effectively harnessing their extensive knowledge becomes paramount. This survey concentrates on the process of distilling the intricate, often implicit knowledge contained within these models into a more symbolic, explicit form. This transformation is crucial for enhancing the interpretability, efficiency, and applicability of LLMs. We categorize the existing research based on methodologies and applications, focusing on how symbolic knowledge distillation can be used to improve the transparency and functionality of smaller, more efficient Artificial Intelligence (AI) models. The survey discusses the core challenges, including maintaining the depth of knowledge in a comprehensible format, and explores the various approaches and techniques that have been developed in this field. We identify gaps in current research and potential opportunities for future advancements. This survey aims to provide a comprehensive overview of symbolic knowledge distillation in LLMs, spotlighting its significance in the progression towards more accessible and efficient AI systems.

A Survey on Symbolic Knowledge Distillation of Large Language Models

TL;DR

This survey maps the growing field of symbolic knowledge distillation for large language models (LLMs), defining symbolic KD as translating implicit neural knowledge into explicit symbolic representations such as rules and graphs. It categorizes approaches into Direct, Multilevel, and RL-based distillation, and reviews how these methods interact with LLM architectures, training paradigms, and downstream applications. The paper synthesizes related works on knowledge bases, consistency, editing, reasoning, and explainability, highlighting advances in commonsense, translation, summarisation, mathematics, vision-language, and instruction generation. It also discusses opportunities, challenges, and practical implications, emphasizing data quality, human-machine collaboration, and the potential for open-source data and models. Overall, the survey argues that symbolic KD offers a viable, scalable path to more interpretable, efficient, and domain-tailored AI systems that complement ever-larger LLMs.

Abstract

This survey paper delves into the emerging and critical area of symbolic knowledge distillation in Large Language Models (LLMs). As LLMs like Generative Pre-trained Transformer-3 (GPT-3) and Bidirectional Encoder Representations from Transformers (BERT) continue to expand in scale and complexity, the challenge of effectively harnessing their extensive knowledge becomes paramount. This survey concentrates on the process of distilling the intricate, often implicit knowledge contained within these models into a more symbolic, explicit form. This transformation is crucial for enhancing the interpretability, efficiency, and applicability of LLMs. We categorize the existing research based on methodologies and applications, focusing on how symbolic knowledge distillation can be used to improve the transparency and functionality of smaller, more efficient Artificial Intelligence (AI) models. The survey discusses the core challenges, including maintaining the depth of knowledge in a comprehensible format, and explores the various approaches and techniques that have been developed in this field. We identify gaps in current research and potential opportunities for future advancements. This survey aims to provide a comprehensive overview of symbolic knowledge distillation in LLMs, spotlighting its significance in the progression towards more accessible and efficient AI systems.
Paper Structure (57 sections, 7 figures, 4 tables)

This paper contains 57 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Milestones in Knowledge Distillation and Large Language Models
  3. Background and Preliminaries
  4. Knowledge Distillation
  5. Response-based Knowledge Distillation
  6. Feature-based Knowledge Distillation
  7. Relation-based Knowledge Distillation
  8. Symbolic Knowledge Distillation
  9. Large Language Models
  10. Architecture
  11. Training Process of Large Language Models
  12. Symbolic Knowledge Distillation of Large Language Models
  13. Direct Distillation
  14. Multilevel distillation of symbolic knowledge
  15. Distillation using Reinforcement Learning policy
  16. ...and 42 more sections

Figures (7)

  • Figure 1: Milestones in history of LLM and Knowledge Distillation
  • Figure 2: Types of Traditional Knowledge Distillation (a) Response-based, (b) Feature-based and (c) Relation-based
  • Figure 3: Symbolic Knowledge Distillation
  • Figure 4: Overview of Direct Distillation process LLMs
  • Figure 5: Overview of Multilevel Distillation process LLMs
  • ...and 2 more figures