Domain Specific Specialization in Low-Resource Settings: The Efficacy of Offline Response-Based Knowledge Distillation in Large Language Models

TL;DR

This work tackles hallucinations in domain-specific knowledge by proposing an offline response-based knowledge distillation pipeline to create a Regulation Expert for Düzce University under hardware constraints. It compares three data strategies and shows that a 500-line context-aware dataset achieves 96.7% accuracy, while larger datasets fail due to lack of grounding. Hardware efficiency is achieved via Unsloth and 4-bit QLoRA, enabling 7B models to train with 16 GB VRAM and faster training. The approach delivers robust rejection capabilities (100% in adversarial cases) and highlights that data quality and contextual structure are more critical than sheer size for domain adaptation in low-resource settings.

Abstract

Large Language Models (LLMs) excel in general tasks but often struggle with hallucinations when handling domain-specific or institutional knowledge absent from their pre-training. We present an offline response-based knowledge distillation method that develops high-accuracy specialized assistants under constrained hardware resources. We evaluate three distinct data strategies: general domain adaptation (15,000 lines), unstructured knowledge injection (2,000 lines), and a context-aware synthetic dataset (500 lines) generated by a teacher model. To minimize computational costs, we utilize the Unsloth library to optimize the Qwen-2.5-7B student model, reducing NVIDIA A100 GPU memory requirements from 40 GB to 16 GB. Experimental results demonstrate that while larger unstructured datasets suffer from persistent hallucinations, the 500-line context-aware dataset achieves a 96.7% accuracy rate and robust rejection capability. These findings validate the LIMA hypothesis, showing that data quality and structural alignment are more critical than quantity for domain adaptation in low-resource settings.