Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoder-based Sense Knowledge Distillation

Qitong Wang, Mohammed J. Zaki, Georgios Kollias, Vasileios Kalantzis

Abstract

Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.

Decoder-based Sense Knowledge Distillation

Abstract

Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.
Paper Structure (41 sections, 9 equations, 3 figures, 12 tables)

This paper contains 41 sections, 9 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Decoder-only language models:
  4. Sense Embeddings:
  5. Knowledge distillation:
  6. DSKD Sense Dictionary
  7. Token-level Sense Dictionary Construction
  8. Lexical Relationship Construction
  9. Word-level Sense Composition
  10. DSKD Methodology
  11. Conceptual Overview
  12. Standard autoregressive language modeling.
  13. Knowledge Distillation.
  14. Decoder-based Sense Knowledge Distillation.
  15. Training Progress
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Distribution of selected word embeddings from Llama-3-8B-Instruct dubey2024llama on Wikipedia dump wikipedia_dump_20240320, with t-SNE van2008visualizing for visualization. Semantically related tokens form coherent clusters in the embedding space: synonyms of red (e.g., ruby, crimson, vermilion, scarlet) appear in a compact region, while vehicle-related tokens such as bus and car occupy a separate neighborhood. For words that are decomposed into multiple subword tokens by the tokenizer (e.g., vermilion, scarlet), a composed representation is used for visualization; details are provided in Section 3.1. This structured organization suggests that contextual embeddings encode discrete semantic regularities.
  • Figure 2: Overview of DSKD training: Given an input corpus, the teacher's last hidden layer $\mathbf{m}_t$ is used to retrieve sense embeddings of the token $t+1$ and its synonyms (positive candidates) and antonyms (negative candidates) from the sense dictionary. The $\kappa$ nearest positive and negative embeddings to $\mathbf{m}_t$ are selected, and the student's last hidden layer $\mathbf{n}_t$ is optimized with a hinge MSE loss. This process aligns the student’s semantic space with the teacher's sense dictionary.
  • Figure 3: Performance of KD and DSKD distilled from Llama-3-8b-Instruct on ARC and MMLU across varying (a) total number of layers, and (b) number of trainable student layers.