ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs

TL;DR

ODA introduces an Observation-Driven Agent that tightly couples knowledge-graph reasoning with large language models through an observation–action–reflection loop. By constructing recursive KG observations and integrating them into action and reflection, ODA achieves state-of-the-art performance on four KBQA benchmarks, with substantial gains over prompt-based and fine-tuned baselines. The approach demonstrates robustness across backbones (GPT-3.5, GPT-4, DeepSeek-V2) and highlights the practical value of KG-guided observation for complex reasoning tasks. This work advances KBQA by providing a scalable, autonomous mechanism to leverage rich KG patterns within LLM-driven reasoning.

Abstract

The integration of Large Language Models (LLMs) and knowledge graphs (KGs) has achieved remarkable success in various natural language processing tasks. However, existing methodologies that integrate LLMs and KGs often navigate the task-solving process solely based on the LLM's analysis of the question, overlooking the rich cognitive potential inherent in the vast knowledge encapsulated in KGs. To address this, we introduce Observation-Driven Agent (ODA), a novel AI agent framework tailored for tasks involving KGs. ODA incorporates KG reasoning abilities via global observation, which enhances reasoning capabilities through a cyclical paradigm of observation, action, and reflection. Confronting the exponential explosion of knowledge during observation, we innovatively design a recursive observation mechanism. Subsequently, we integrate the observed knowledge into the action and reflection modules. Through extensive experiments, ODA demonstrates state-of-the-art performance on several datasets, notably achieving accuracy improvements of 12.87% and 8.9%.

Figures (3)

• Figure 1: An example of LLM integrating with KG. Observed entities are shown in white, while non-observed entities are displayed in gray. Entities selected by the agent to answer the question are highlighted in yellow.
• Figure 2: The overall framework of $\textsc{ODA}$.
• Figure 3: An example workflow of $\textsc{ODA}$. In this case, $\textsc{ODA}$ initiates the obervation with entity Johann Wolfgang von Goethe. During the first iteration on the left side, the Neighbor Exploration of Johann Wolfgang von Goethe is selected, and the reflected triple (Johann Wolfgang von Goethe, unmarried Partner, Lili Schöneman) is stored in memory. Subsequently, The observation of Lili Schöneman then guides $\textsc{ODA}$ to choose Neighbor Exploration action, and leads to the retention of the triple (Lili Schöneman, place of birth, Offenbach am Main) in memory, as shown on the right side. Once sufficient knowledge has been accumulated, $\textsc{ODA}$ triggers the Answer action, correctly identifying Offenbach am Main as the answer.