Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data
Xue Wu, Kostas Tsioutsiouliklis
TL;DR
This work tackles hallucination and complex multi-hop reasoning in LLMs by grounding reasoning in knowledge graphs encoded as programming language constructs, notably Python. By comparing natural language, JSON, and Python representations, and applying LoRA-based fine-tuning to Llama-3.1-8B-Instruct, the authors show that Python-based KG encodings yield superior multi-hop reasoning performance and better generalization to longer reasoning paths. Fine-tuned models with Python representations can even surpass much larger baselines in context-rich or three-hop scenarios, underscoring the value of tightly integrated, code-level KG representations for grounded reasoning. The results suggest a practical path to reduce hallucinations and enhance interpretability by encoding and guiding KG-driven inference within the model's own reasoning process.
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, they often struggle with complex reasoning tasks and are prone to hallucination. Recent research has shown promising results in leveraging knowledge graphs (KGs) to enhance LLM performance. KGs provide a structured representation of entities and their relationships, offering a rich source of information that can enhance the reasoning capabilities of LLMs. For this work, we have developed different techniques that tightly integrate KG structures and semantics into LLM representations. Our results show that we are able to significantly improve the performance of LLMs in complex reasoning scenarios, and ground the reasoning process with KGs. We are the first to represent KGs with programming language and fine-tune pretrained LLMs with KGs. This integration facilitates more accurate and interpretable reasoning processes, paving the way for more advanced reasoning capabilities of LLMs.