UniOQA: A Unified Framework for Knowledge Graph Question Answering with Large Language Models

TL;DR

This work tackles KGQA over the large Chinese OwnThink knowledge graph by introducing UniOQA, a unified framework that runs two parallel workflows: a Translator that fine-tunes an LLM to generate executable Cypher queries (CQL) and a Searcher that employs Retrieval-Augmented Generation (GRAG) to retrieve direct answers. The Translator is augmented by an Entity and Relation Replacement step to align CQL with the KG, while the Searcher provides complementary answers from retrieved subgraphs; final answers are produced via a Dynamic Decision Algorithm that fuses both sources. Empirical results on SpCQL show state-of-the-art performance with ACC_LX 21.2% and ACC_EX 54.9%, outperforming all baselines, and ablation studies attribute gains to improved representation (ERR, Neft) and the effectiveness of combining translation and retrieval. The approach demonstrates the value of integrating precise CQL generation with retrieval-based supplementation, offering practical improvements for KGQA and guiding future work on unstructured data integration and efficiency optimization.

Abstract

OwnThink stands as the most extensive Chinese open-domain knowledge graph introduced in recent times. Despite prior attempts in question answering over OwnThink (OQA), existing studies have faced limitations in model representation capabilities, posing challenges in further enhancing overall accuracy in question answering. In this paper, we introduce UniOQA, a unified framework that integrates two complementary parallel workflows. Unlike conventional approaches, UniOQA harnesses large language models (LLMs) for precise question answering and incorporates a direct-answer-prediction process as a cost-effective complement. Initially, to bolster representation capacity, we fine-tune an LLM to translate questions into the Cypher query language (CQL), tackling issues associated with restricted semantic understanding and hallucinations. Subsequently, we introduce the Entity and Relation Replacement algorithm to ensure the executability of the generated CQL. Concurrently, to augment overall accuracy in question answering, we further adapt the Retrieval-Augmented Generation (RAG) process to the knowledge graph. Ultimately, we optimize answer accuracy through a dynamic decision algorithm. Experimental findings illustrate that UniOQA notably advances SpCQL Logical Accuracy to 21.2% and Execution Accuracy to 54.9%, achieving the new state-of-the-art results on this benchmark. Through ablation experiments, we delve into the superior representation capacity of UniOQA and quantify its performance breakthrough.

Figures (5)

• Figure 1: The overview of our framework. We construct it with two parallel workflows: (1) Translator (Section \ref{['sec:ftllm']}&\ref{['sec:algr']}), which consists of fine-tuning LLM for CQL generation and modifying the entities and relations in CQL. (2) Searcher (Section \ref{['sec:grag']}), which employs a direct search approach within the knowledge graph to retrieve answers relevant to the posed questions. Finally, the answers from the two workflows undergo optimization via the dynamic decision algorithm (Section \ref{['sec:comb']}), yielding the ultimate results. The example in this figure have been thoroughly derived in Section \ref{['sec:met']}.
• Figure 2: The overview of GRAG. Also the overview of the Searcher workflow.
• Figure 3: Qualitative examples from test results. The red areas indicate the parts that lead to errors. The original text is in Chinese, and we have translated the question, answer, and entities and relations into English.
• Figure 4: (a) Performance of models trained on different scales (datasize & step); (b) Performance of UniOQA as the decision operator $\sigma$ changes.
• Figure 5: (a) Distribution of results across unique workflows; (b) Performance comparison of different workflows under varying difficulty levels. In the two figures, SP refers to the Translator workflow, and IR refers to the Searcher workflow.