Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs

Minh-Vuong Nguyen, Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, Gholamreza Haffari

TL;DR

The paper tackles the challenge of evaluating chain-of-thought in multi-hop knowledge-graph QA by grounding CoT in KG paths. It introduces discriminative and generative evaluation modules to separately assess whether LLMs know faithful reasoning and whether they can produce faithful CoT when prompted. Across five LLM families and two complex datasets (CWQ, GrailQA), the authors find that while models hold reasoning knowledge, CoT faithfulness often lags behind answer accuracy, with the gap tending to widen for larger models; prompting improvements like planning and self-consistency help both answer quality and CoT faithfulness. Ground-truth reasoning paths derived from SPARQL-converted Freebase graphs enable automatic, structured evaluation of CoT arguments, offering a path toward more interpretable and reliable reasoning from LLMs. The framework highlights the importance of evaluating reasoning steps themselves, not just final answers, and provides a scalable approach for future research on faithful AI reasoning across domains.

Abstract

Large language models (LLMs) demonstrate strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs' knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that LLMs possess sufficient knowledge to perform reasoning. However, there exists a significant disparity between answer accuracy and faithfulness of the CoT reasoning generated by LLMs, indicating that they often arrive at correct answers through incorrect reasoning.

Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs

TL;DR

The paper tackles the challenge of evaluating chain-of-thought in multi-hop knowledge-graph QA by grounding CoT in KG paths. It introduces discriminative and generative evaluation modules to separately assess whether LLMs know faithful reasoning and whether they can produce faithful CoT when prompted. Across five LLM families and two complex datasets (CWQ, GrailQA), the authors find that while models hold reasoning knowledge, CoT faithfulness often lags behind answer accuracy, with the gap tending to widen for larger models; prompting improvements like planning and self-consistency help both answer quality and CoT faithfulness. Ground-truth reasoning paths derived from SPARQL-converted Freebase graphs enable automatic, structured evaluation of CoT arguments, offering a path toward more interpretable and reliable reasoning from LLMs. The framework highlights the importance of evaluating reasoning steps themselves, not just final answers, and provides a scalable approach for future research on faithful AI reasoning across domains.

Abstract

Large language models (LLMs) demonstrate strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs' knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that LLMs possess sufficient knowledge to perform reasoning. However, there exists a significant disparity between answer accuracy and faithfulness of the CoT reasoning generated by LLMs, indicating that they often arrive at correct answers through incorrect reasoning.
Paper Structure (58 sections, 3 equations, 13 figures, 8 tables, 3 algorithms)

This paper contains 58 sections, 3 equations, 13 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Chain-of-thought (CoT) Reasoning
  4. Faithful CoT
  5. Knowledge Graphs (KGs)
  6. Reasoning Paths
  7. Faithful CoT Grounded by KGs
  8. Evaluating the CoT Reasoning of LLMs
  9. Discriminative Evaluation
  10. Findings
  11. Generative Evaluation
  12. Reasoning Path Construction
  13. Reasoning Path Evaluation
  14. Validity of Reasoning Path
  15. Fine-grained Assessment
  16. ...and 43 more sections

Figures (13)

  • Figure 1: Examples of different reasoning errors and a faithful CoT grounded by knowledge graph.
  • Figure 2: Discriminative Evaluation Prompt. <Question> indicates the question, <Answer> denotes the corresponding answer, and <Reasoning Path> denotes the input reasoning path, which is verbalized as a structured sentence
  • Figure 3: The overall framework of evaluating the CoT reasoning of LLMs, which contains two evaluation modules: discriminative evaluation and generative evaluation. The orange and red rectangles denote the entities mentioned in the question and answer, respectively.
  • Figure 4: The breakdown of reasoning error types in CWQ and GrailQA.
  • Figure 5: The precision of LLMs using few-shot CoT prompt.
  • ...and 8 more figures

Theorems & Definitions (2)

  • Example 1
  • Example 2