Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization

TL;DR

SMART-SLIC addresses the brittleness of general LLMs in domain-specific QA by integrating retrieval-augmented generation with a domain-specific knowledge graph and a vector store, both constructed without LLM-driven data generation. The framework leverages nonnegative tensor factorization to derive latent topics and ontology features, which are encoded into a Neo4j KG and complemented by a Milvus vector store of document content and paragraphs. The retrieval pipeline uses ReAct agents and graph-guided queries (including PROFILE-aided synthesized queries) to produce source-attributed, reliable answers, demonstrated on malware analysis and anomaly detection literature with substantial performance gains over non-RAG baselines. The work shows strong potential for generalizing to other specialized domains and points to future improvements in graph completion, entity linking, and cross-domain applicability.

Abstract

Large Language Models (LLMs) are pre-trained on large-scale corpora and excel in numerous general natural language processing (NLP) tasks, such as question answering (QA). Despite their advanced language capabilities, when it comes to domain-specific and knowledge-intensive tasks, LLMs suffer from hallucinations, knowledge cut-offs, and lack of knowledge attributions. Additionally, fine tuning LLMs' intrinsic knowledge to highly specific domains is an expensive and time consuming process. The retrieval-augmented generation (RAG) process has recently emerged as a method capable of optimization of LLM responses, by referencing them to a predetermined ontology. It was shown that using a Knowledge Graph (KG) ontology for RAG improves the QA accuracy, by taking into account relevant sub-graphs that preserve the information in a structured manner. In this paper, we introduce SMART-SLIC, a highly domain-specific LLM framework, that integrates RAG with KG and a vector store (VS) that store factual domain specific information. Importantly, to avoid hallucinations in the KG, we build these highly domain-specific KGs and VSs without the use of LLMs, but via NLP, data mining, and nonnegative tensor factorization with automatic model selection. Pairing our RAG with a domain-specific: (i) KG (containing structured information), and (ii) VS (containing unstructured information) enables the development of domain-specific chat-bots that attribute the source of information, mitigate hallucinations, lessen the need for fine-tuning, and excel in highly domain-specific question answering tasks. We pair SMART-SLIC with chain-of-thought prompting agents. The framework is designed to be generalizable to adapt to any specific or specialized domain. In this paper, we demonstrate the question answering capabilities of our framework on a corpus of scientific publications on malware analysis and anomaly detection.

Figures (7)

• Figure 1: User query routing overview.
• Figure 2: The RAG pipeline. Images generated with DALL·E dalle_tensor_decomp_arxiv_images.
• Figure 3: User query routing overview.
• Figure 4: Nodes and tools of the ReAct agent. Images from DALL·E dalle_tensor_decomp_arxiv_images.
• Figure 5: The KG schema. Images generated with DALL·E dalle_tensor_decomp_arxiv_images.