Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom

Rishika Sen, Sujoy Roychowdhury, Sumit Soman, H. G. Ranjani, Srikhetra Mohanty

TL;DR

The paper tackles domain adaptation for telecom-domain QA by studying knowledge distillation between teacher and student LLMs. It systematically varies supervised fine-tuning (SFT) on the teacher, on the student, or both, and contrasts vanilla KD with Dual Space KD (DSKD) across matched and mismatched vocabularies using the TeleQuAD dataset and 14 diverse evaluation metrics. Key findings show that SFT of the teacher improves distilled performance when vocabularies match, and that training both teacher and student yields the best overall results, with vocabulary and algorithm choices modulating gains. The work provides practical guidance for domain-adaptive KD and suggests directions for scaling to larger models and MoEs, as well as extending to other tasks and evaluation frameworks.

Abstract

Knowledge Distillation (KD) is one of the approaches to reduce the size of Large Language Models (LLMs). A LLM with smaller number of model parameters (student) is trained to mimic the performance of a LLM of a larger size (teacher model) on a specific task. For domain-specific tasks, it is not clear if teacher or student model, or both, must be considered for domain adaptation. In this work, we study this problem from perspective of telecom domain Question-Answering (QA) task. We systematically experiment with Supervised Fine-tuning (SFT) of teacher only, SFT of student only and SFT of both prior to KD. We design experiments to study the impact of vocabulary (same and different) and KD algorithms (vanilla KD and Dual Space KD, DSKD) on the distilled model. Multi-faceted evaluation of the distillation using 14 different metrics (N-gram, embedding and LLM-based metrics) is considered. Experimental results show that SFT of teacher improves performance of distilled model when both models have same vocabulary, irrespective of algorithm and metrics. Overall, SFT of both teacher and student results in better performance across all metrics, although the statistical significance of the same depends on the vocabulary of the teacher models.

Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom

TL;DR

The paper tackles domain adaptation for telecom-domain QA by studying knowledge distillation between teacher and student LLMs. It systematically varies supervised fine-tuning (SFT) on the teacher, on the student, or both, and contrasts vanilla KD with Dual Space KD (DSKD) across matched and mismatched vocabularies using the TeleQuAD dataset and 14 diverse evaluation metrics. Key findings show that SFT of the teacher improves distilled performance when vocabularies match, and that training both teacher and student yields the best overall results, with vocabulary and algorithm choices modulating gains. The work provides practical guidance for domain-adaptive KD and suggests directions for scaling to larger models and MoEs, as well as extending to other tasks and evaluation frameworks.

Abstract

Knowledge Distillation (KD) is one of the approaches to reduce the size of Large Language Models (LLMs). A LLM with smaller number of model parameters (student) is trained to mimic the performance of a LLM of a larger size (teacher model) on a specific task. For domain-specific tasks, it is not clear if teacher or student model, or both, must be considered for domain adaptation. In this work, we study this problem from perspective of telecom domain Question-Answering (QA) task. We systematically experiment with Supervised Fine-tuning (SFT) of teacher only, SFT of student only and SFT of both prior to KD. We design experiments to study the impact of vocabulary (same and different) and KD algorithms (vanilla KD and Dual Space KD, DSKD) on the distilled model. Multi-faceted evaluation of the distillation using 14 different metrics (N-gram, embedding and LLM-based metrics) is considered. Experimental results show that SFT of teacher improves performance of distilled model when both models have same vocabulary, irrespective of algorithm and metrics. Overall, SFT of both teacher and student results in better performance across all metrics, although the statistical significance of the same depends on the vocabulary of the teacher models.
Paper Structure (25 sections, 3 figures, 3 tables)

This paper contains 25 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem statement
  3. Overview of KD techniques
  4. Vocabulary
  5. KD Algorithms
  6. Overview of metrics
  7. Contributions
  8. Methodology
  9. Experimental Setup
  10. Hypothesis tests
  11. Dataset description
  12. Environment
  13. Experimental Results
  14. Impact of SFT
  15. Llama
  16. ...and 10 more sections

Figures (3)

  • Figure 1: A schematic representation of experiments consisting of the choice of SFT for teacher student, the choice of distillation algorithms, Vanilla or DSKD, and choice of evaluation metrics.
  • Figure 2: Schematic representation of different choices based on which we conduct Hypothesis tests
  • Figure 4: Group-wise average of performance metrics from the heatmap in Fig. \ref{['pic:heatmap']}.