Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph

TL;DR

Way-to-Specialist (WTS) tackles the challenge of domain-specific reasoning without specialized training by creating a bidirectional loop between a domain knowledge graph (DKG) and a specialized LLM. The framework combines retrieval-augmented generation with dynamic KG evolution, comprising DKG-Augmented LLM for iterative knowledge retrieval and reasoning, and LLM-Assisted DKG Evolution for generating new KG triples to enrich the DKG. Evaluations across six QA datasets in five domains show WTS delivering state-of-the-art gains in four domains, with a maximum improvement of 11.3%, and demonstrate the value of an evolving, personalized DKГ for robust domain specialization. The approach avoids re-training costs and offers practical implications for building continuously improving, domain-aware AI systems in real-world applications.

Abstract

Large language models (LLMs) have demonstrated exceptional performance across a wide variety of domains. Nonetheless, generalist LLMs continue to fall short in reasoning tasks necessitating specialized knowledge. Prior investigations into specialized LLMs focused on domain-specific training, which entails substantial efforts in domain data acquisition and model parameter fine-tuning. To address these challenges, this paper proposes the Way-to-Specialist (WTS) framework, which synergizes retrieval-augmented generation with knowledge graphs (KGs) to enhance the specialized capability of LLMs in the absence of specialized training. In distinction to existing paradigms that merely utilize external knowledge from general KGs or static domain KGs to prompt LLM for enhanced domain-specific reasoning, WTS proposes an innovative "LLM$\circlearrowright$KG" paradigm, which achieves bidirectional enhancement between specialized LLM and domain knowledge graph (DKG). The proposed paradigm encompasses two closely coupled components: the DKG-Augmented LLM and the LLM-Assisted DKG Evolution. The former retrieves question-relevant domain knowledge from DKG and uses it to prompt LLM to enhance the reasoning capability for domain-specific tasks; the latter leverages LLM to generate new domain knowledge from processed tasks and use it to evolve DKG. WTS closes the loop between DKG-Augmented LLM and LLM-Assisted DKG Evolution, enabling continuous improvement in the domain specialization as it progressively answers and learns from domain-specific questions. We validate the performance of WTS on 6 datasets spanning 5 domains. The experimental results show that WTS surpasses the previous SOTA in 4 specialized domains and achieves a maximum performance improvement of 11.3%.

Figures (8)

• Figure 1: Comparison of "LLM$\circlearrowright$KG" against SOTA paradigms in KG-augmented LLM.
• Figure 2: Overview of WTS. The upper part is DKG-Augmented LLM which extracts entities from the received question and performs iterative Retrieval-Prune-Reason processes. The bottom part is LLM-Assisted DKG Evolution which generates new knowledge triples from the processed question to evolve the DKG.
• Figure 3: Illustration of WTS formation pipeline.
• Figure 4: Retrieval time and execution time of WTS with GPT-3.5.
• Figure 5: DKG Size of WTS with GPT-3.5 and GPT-4o.