UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG

Abstract

Large language models (LLMs) frequently generate confident yet factually incorrect content when used for language generation (a phenomenon often known as hallucination). Retrieval augmented generation (RAG) tries to reduce factual errors by identifying information in a knowledge corpus and putting it in the context window of the model. While this approach is well-established for document-structured data, it is non-trivial to adapt it for Knowledge Graphs (KGs), especially for queries that require multi-node/multi-hop reasoning on graphs. We introduce ULTRAG, a general framework for retrieving information from Knowledge Graphs that shifts away from classical RAG. By endowing LLMs with off-the-shelf neural query executing modules, we highlight how readily available language models can achieve state-of-the-art results on Knowledge Graph Question Answering (KGQA) tasks without any retraining of the LLM or executor involved. In our experiments, ULTRAG achieves better performance when compared to state-of-the-art KG-RAG solutions, and it enables language models to interface with Wikidata-scale graphs (116M entities, 1.6B relations) at comparable or lower costs.

Figures (4)

• Figure 1: UltRAG pipeline. The LLM is provided with the syntactic rules for queries and the relation types. Ground-truth seed entities (Turing Award, Deep Learning, etc.) may be given, but, if not, an entity linking step takes place. The generated query is then neurally executed against the knowledge graph (each node receives a probability to be an answer at this stage). The most likely query answers are fed back to the LLM, which weighs both the returned probabilities and the semantic meaning of entities, and produces a final answer set.
• Figure 2: Haskell-like grammar definitions for the old BetaE format (left) and our preferred DSL (right). The former uses nested tuples for projections and (binary) intersections. Our DSL uses infix notation with -> for projections and $n$-ary tuples for intersections, attempting to reduce bracket nesting. Below the horizontal line we show how the example from \ref{['fig:vis_abstract']} would transform (entities have been abbreviated). The maximum nesting depth reduces from 4 to 1.
• Figure 3: Comparison of UltraQuery vs symbolic query execution on GTSQA. Both receive identical queries generated by the LLM. Number inside brackets denotes projections, concatenation of expression denotes intersection. Best viewed on screen.
• Figure 4: BetaE format representation of the Turing Award query example from \ref{['fig:vis_abstract']} (entities have been abbreviated). Projections follow the language defined in \ref{['fig:query_language_comparison']}, left. The LLM is tasked with producing the full query; the breakdown is added only for human readability.