Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Learning Content Retrieval with Knowledge Injection

Batuhan Sariturk, Rabia Bayraktar, Merve Elmas Erdem

TL;DR

The paper addresses the challenge of building an efficient, domain-specific educational chatbot for ICT by marrying Retrieval Augmented Generation with parameter-efficient fine-tuning of compact Phi SLMs. It constructs a vector database from Huawei Talent Platform content and contrasts RAG-enabled Phi-2/Phi-3 against fine-tuned baselines using BLEU, ROUGE, METEOR, and BERTScore. Key findings show that a RAG-enabled Phi-2 system achieves strong grounding and semantic fidelity, while fine-tuning provides robust intra-domain accuracy; combining both approaches yields the best overall performance on several metrics. The work demonstrates a practical, resource-conscious workflow for deploying domain-specific educational chatbots and highlights the value of external knowledge retrieval in keeping responses current and relevant, with potential for scalable deployment across STEM education platforms.

Abstract

With the rise of online education platforms, there is a growing abundance of educational content across various domain. It can be difficult to navigate the numerous available resources to find the most suitable training, especially in domains that include many interconnected areas, such as ICT. In this study, we propose a domain-specific chatbot application that requires limited resources, utilizing versions of the Phi language model to help learners with educational content. In the proposed method, Phi-2 and Phi-3 models were fine-tuned using QLoRA. The data required for fine-tuning was obtained from the Huawei Talent Platform, where courses are available at different levels of expertise in the field of computer science. RAG system was used to support the model, which was fine-tuned by 500 Q&A pairs. Additionally, a total of 420 Q&A pairs of content were extracted from different formats such as JSON, PPT, and DOC to create a vector database to be used in the RAG system. By using the fine-tuned model and RAG approach together, chatbots with different competencies were obtained. The questions and answers asked to the generated chatbots were saved separately and evaluated using ROUGE, BERTScore, METEOR, and BLEU metrics. The precision value of the Phi-2 model supported by RAG was 0.84 and the F1 score was 0.82. In addition to a total of 13 different evaluation metrics in 4 different categories, the answers of each model were compared with the created content and the most appropriate method was selected for real-life applications.

Efficient Learning Content Retrieval with Knowledge Injection

TL;DR

The paper addresses the challenge of building an efficient, domain-specific educational chatbot for ICT by marrying Retrieval Augmented Generation with parameter-efficient fine-tuning of compact Phi SLMs. It constructs a vector database from Huawei Talent Platform content and contrasts RAG-enabled Phi-2/Phi-3 against fine-tuned baselines using BLEU, ROUGE, METEOR, and BERTScore. Key findings show that a RAG-enabled Phi-2 system achieves strong grounding and semantic fidelity, while fine-tuning provides robust intra-domain accuracy; combining both approaches yields the best overall performance on several metrics. The work demonstrates a practical, resource-conscious workflow for deploying domain-specific educational chatbots and highlights the value of external knowledge retrieval in keeping responses current and relevant, with potential for scalable deployment across STEM education platforms.

Abstract

With the rise of online education platforms, there is a growing abundance of educational content across various domain. It can be difficult to navigate the numerous available resources to find the most suitable training, especially in domains that include many interconnected areas, such as ICT. In this study, we propose a domain-specific chatbot application that requires limited resources, utilizing versions of the Phi language model to help learners with educational content. In the proposed method, Phi-2 and Phi-3 models were fine-tuned using QLoRA. The data required for fine-tuning was obtained from the Huawei Talent Platform, where courses are available at different levels of expertise in the field of computer science. RAG system was used to support the model, which was fine-tuned by 500 Q&A pairs. Additionally, a total of 420 Q&A pairs of content were extracted from different formats such as JSON, PPT, and DOC to create a vector database to be used in the RAG system. By using the fine-tuned model and RAG approach together, chatbots with different competencies were obtained. The questions and answers asked to the generated chatbots were saved separately and evaluated using ROUGE, BERTScore, METEOR, and BLEU metrics. The precision value of the Phi-2 model supported by RAG was 0.84 and the F1 score was 0.82. In addition to a total of 13 different evaluation metrics in 4 different categories, the answers of each model were compared with the created content and the most appropriate method was selected for real-life applications.

Paper Structure

This paper contains 20 sections, 12 equations, 12 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Question-Answering Systems
  4. Fine-tuning, Large Language Models (LLMs)
  5. Retrieval Augmented Generation (RAG)
  6. Materials and Method
  7. Dataset
  8. Q&A pairs
  9. PDF/PPT Information Extraction
  10. Parameter Efficient Fine-Tuning
  11. Retrieval-Augmented Generation
  12. RAG vs Fine-tuning
  13. Proposed Method
  14. Evaluation Metrics
  15. Experiment and Results
  16. ...and 5 more sections

Figures (12)

  • Figure 1: Path for study and exam
  • Figure 2: Q&A pair examples from generated datasets
  • Figure 3: A Simple Representation of QLoRA
  • Figure 4: General structure of a RAG system
  • Figure 5: Proposed Method
  • ...and 7 more figures