Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations

TL;DR

This work addresses the challenge of generating precise, contextually grounded explanations for learning recommendations by integrating a knowledge graph (KG) as contextual information into prompts for GPT-4. The KG is built from educational materials aligned to a four-level taxonomy, with semantic relations and metadata extracted to provide structured context, guiding a pedagogically informed prompt design. Explanations are produced via a two-part GPT-4 prompt and delivered through a conversational interface, with domain experts shaping the contextual guidelines and explanation templates. Evaluation combining Rouge metrics and qualitative feedback shows that KG-contextualized explanations achieve higher precision/recall and reduce irrelevant content compared to non-contextualized outputs, albeit with limitations related to sample size, cross-LLM comparisons, and user-data personalization under privacy constraints. The approach demonstrates a practical pathway to more reliable, explainable learning recommendations using KG-informed prompts and expert-driven pedagogy.

Abstract

In the era of personalized education, the provision of comprehensible explanations for learning recommendations is of a great value to enhance the learner's understanding and engagement with the recommended learning content. Large language models (LLMs) and generative AI in general have recently opened new doors for generating human-like explanations, for and along learning recommendations. However, their precision is still far away from acceptable in a sensitive field like education. To harness the abilities of LLMs, while still ensuring a high level of precision towards the intent of the learners, this paper proposes an approach to utilize knowledge graphs (KG) as a source of factual context, for LLM prompts, reducing the risk of model hallucinations, and safeguarding against wrong or imprecise information, while maintaining an application-intended learning context. We utilize the semantic relations in the knowledge graph to offer curated knowledge about learning recommendations. With domain-experts in the loop, we design the explanation as a textual template, which is filled and completed by the LLM. Domain experts were integrated in the prompt engineering phase as part of a study, to ensure that explanations include information that is relevant to the learner. We evaluate our approach quantitatively using Rouge-N and Rouge-L measures, as well as qualitatively with experts and learners. Our results show an enhanced recall and precision of the generated explanations compared to those generated solely by the GPT model, with a greatly reduced risk of generating imprecise information in the final learning explanation.

Figures (3)

• Figure 1: Structural information added to the LLM context from the KG. Top: learning path as an output of the recommendation system. Bottom: recommended path as it appears in the KG. Area (A): hierarchical structure of the learning goal. Area (B) KG community around LO3 and LO4. Connection (C): semantic relation extracted by the relation extraction algorithm.
• Figure 2: Proposed approach for constructing the GPT-4 prompt, with KG-based contextualization, as well as the Chatbot-based user interaction, and the expert roles in the design for context and explanation-templates.
• Figure 3: Recall, precision, and f1-measure values of the Rouge metric, for both explanation types: 1) with KG-based contextualization (blue), and 2) without contextualization (gray).