Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs

Elan Markowitz, Anil Ramakrishna, Jwala Dhamala, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

TL;DR

Tree-of-Traversals addresses the problem of augmenting black-box LLMs with knowledge graphs in a zero-shot setting without training. It introduces a KG interface, a finite-state action machine, and a tree-search procedure that expands a local KG subgraph guided by a learned value function to answer complex, multi-hop questions. The method supports single and multiple KGs, demonstrated on 2WikiMultiHop, QALD-10, and MusicBrainz-x-Wikidata, achieving state-of-the-art zero-shot results on the datasets. The findings show that value-guided exploration and backtracking significantly improve accuracy, and that combining domain- or cross-domain KGs yields substantial gains for knowledge-intensive QA. This work offers a practical, training-free path to robust, up-to-date knowledge augmentation for LLMs with external knowledge sources.

Abstract

Knowledge graphs (KGs) complement Large Language Models (LLMs) by providing reliable, structured, domain-specific, and up-to-date external knowledge. However, KGs and LLMs are often developed separately and must be integrated after training. We introduce Tree-of-Traversals, a novel zero-shot reasoning algorithm that enables augmentation of black-box LLMs with one or more KGs. The algorithm equips a LLM with actions for interfacing a KG and enables the LLM to perform tree search over possible thoughts and actions to find high confidence reasoning paths. We evaluate on two popular benchmark datasets. Our results show that Tree-of-Traversals significantly improves performance on question answering and KG question answering tasks. Code is available at \url{https://github.com/amazon-science/tree-of-traversals}

Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs

TL;DR

Tree-of-Traversals addresses the problem of augmenting black-box LLMs with knowledge graphs in a zero-shot setting without training. It introduces a KG interface, a finite-state action machine, and a tree-search procedure that expands a local KG subgraph guided by a learned value function to answer complex, multi-hop questions. The method supports single and multiple KGs, demonstrated on 2WikiMultiHop, QALD-10, and MusicBrainz-x-Wikidata, achieving state-of-the-art zero-shot results on the datasets. The findings show that value-guided exploration and backtracking significantly improve accuracy, and that combining domain- or cross-domain KGs yields substantial gains for knowledge-intensive QA. This work offers a practical, training-free path to robust, up-to-date knowledge augmentation for LLMs with external knowledge sources.

Abstract

Knowledge graphs (KGs) complement Large Language Models (LLMs) by providing reliable, structured, domain-specific, and up-to-date external knowledge. However, KGs and LLMs are often developed separately and must be integrated after training. We introduce Tree-of-Traversals, a novel zero-shot reasoning algorithm that enables augmentation of black-box LLMs with one or more KGs. The algorithm equips a LLM with actions for interfacing a KG and enables the LLM to perform tree search over possible thoughts and actions to find high confidence reasoning paths. We evaluate on two popular benchmark datasets. Our results show that Tree-of-Traversals significantly improves performance on question answering and KG question answering tasks. Code is available at \url{https://github.com/amazon-science/tree-of-traversals}
Paper Structure (44 sections, 8 figures, 3 tables, 1 algorithm)

This paper contains 44 sections, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Tree-of-Traversals
  3. Knowledge Graph Interface
  4. Action State Machine (ASM)
  5. Prompt Templates.
  6. Invoking KG Interfaces.
  7. Tree Search Algorithm
  8. Value Function Guidance.
  9. Chain-of-Traversals.
  10. Tree-of-Traversals with Multiple KGs
  11. Experiments
  12. Tasks and Datasets.
  13. Metric.
  14. Comparison Baselines.
  15. Implementation details.
  16. ...and 29 more sections

Figures (8)

  • Figure 1: An example of how Tree-of-Traversals uses a KG interface for the query, "What actor played in both Inception and Interstellar?".
  • Figure 2: Action State Machine
  • Figure 3: Prompt for the 'selecting-entities' state of the ASM. The original query and the local KG subgraph are part of the prompt for every action state while the 'Current task' differs. The entity options show what entities can be selected based on the current local KG subgraph. If the model selected Q62519478: Beatrice Stone here, then in the next action state (selecting-relation), the dynamic prompt would list the relation types that Beatrice Stone has edges for.
  • Figure 4: Example question from MusicBrainz-x-Wiki and how Tree-of-Traversals arrives at its final answer. Red indicates entities and and relationships belonging to MusicBrainz. Green indicates relationships only in Wikidata. Blue indicates entities linked to both KGs.
  • Figure 5: The EM-in accuracy for answers with model assigned value 0.0 or 1.0 and the corresponding true EM-in score of the answer. This includes all proposed answers, not just the final answer returned by the model.
  • ...and 3 more figures