Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models

Ge Zhang, Mohammad Ali Alomrani, Hongjian Gu, Jiaming Zhou, Yaochen Hu, Bin Wang, Qun Liu, Mark Coates, Yingxue Zhang, Jianye Hao

TL;DR

Path-of-Thoughts (PoT) introduces a three-stage framework for robust relational reasoning with LLMs by first extracting a task-agnostic graph from the problem narrative, then identifying reasoning paths between query entities, and finally inferring answers via an LLM or a symbolic solver. The graph-centric decomposition mitigates errors from LLM extraction and supports multiple inferential paths, enhancing resilience to ambiguous or conflicting inputs. Across four benchmark datasets, PoT achieves substantial improvements over prompting baselines and neuro-symbolic methods without fine-tuning, particularly on long reasoning chains, and demonstrates robustness to extraction noise. By leveraging both structured graph representations and flexible reasoning backends, PoT offers a scalable, adaptable approach to complex relational reasoning tasks in NLP.

Abstract

Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework designed to tackle relation reasoning by decomposing the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a task-agnostic graph that identifies crucial entities, relations, and attributes within the problem context. Subsequently, PoT identifies relevant reasoning chains within the graph corresponding to the posed question, facilitating inference of potential answers. Experimental evaluations on four benchmark datasets, demanding long reasoning chains, demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (maximum 21.3%) without necessitating fine-tuning or extensive LLM calls. Furthermore, as opposed to prior neuro-symbolic methods, PoT exhibits improved resilience against LLM errors by leveraging the compositional nature of graphs.

Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models

TL;DR

Path-of-Thoughts (PoT) introduces a three-stage framework for robust relational reasoning with LLMs by first extracting a task-agnostic graph from the problem narrative, then identifying reasoning paths between query entities, and finally inferring answers via an LLM or a symbolic solver. The graph-centric decomposition mitigates errors from LLM extraction and supports multiple inferential paths, enhancing resilience to ambiguous or conflicting inputs. Across four benchmark datasets, PoT achieves substantial improvements over prompting baselines and neuro-symbolic methods without fine-tuning, particularly on long reasoning chains, and demonstrates robustness to extraction noise. By leveraging both structured graph representations and flexible reasoning backends, PoT offers a scalable, adaptable approach to complex relational reasoning tasks in NLP.

Abstract

Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework designed to tackle relation reasoning by decomposing the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a task-agnostic graph that identifies crucial entities, relations, and attributes within the problem context. Subsequently, PoT identifies relevant reasoning chains within the graph corresponding to the posed question, facilitating inference of potential answers. Experimental evaluations on four benchmark datasets, demanding long reasoning chains, demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (maximum 21.3%) without necessitating fine-tuning or extensive LLM calls. Furthermore, as opposed to prior neuro-symbolic methods, PoT exhibits improved resilience against LLM errors by leveraging the compositional nature of graphs.

Paper Structure

This paper contains 42 sections, 3 figures, 28 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Related Work
  4. Methodology
  5. Graph Extraction
  6. Structured Prompts with Sectional Markup.
  7. Structured Output with Syntactic Delimiters.
  8. Predefined Output Categories.
  9. Decomposition of the Extraction Task.
  10. Path Identification
  11. Reasoning
  12. LLM Reasoner:
  13. Symbolic Reasoner:
  14. Experiments
  15. Experimental Setup
  16. ...and 27 more sections

Figures (3)

  • Figure 1: The PoT framework with an example featuring robustness against LLM extraction errors highlighted in red (flipped relation). The LLM is employed to extract the graph representing the story's relational structure. Path identification isolates the reasoning paths relevant to the query entities. PoT reasons over each path independently to alleviate cascading errors due to extraction and infer all possible answers.
  • Figure 2: Upper: Accuracy of PoT-Symbolic and LLM-ASP w.r.t noise Types. A: flip -- irrelevant edge, B: add -- new_node -- one new edge, C: add -- new_node -- conflict edge, D: add -- no_node -- irrelevant edge, E: add -- no_node -- main edge, F: replace -- irrelevant edge, and G: disconnected edges. Bottom:: Accuracy of PoT-Symbolic and LLM-ASP w.r.t the number of noises.
  • Figure 3: Illustrations of 7 noise types in the synthetic noise dataset. A toy sample of an instance graph with 5 nodes and 4 edges is shown. The nodes and edges that are relevant or irrelevant to answer the question are marked in red and green, respectively. The noisy nodes/edges are marked in purple.