HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation

TL;DR

HGOT tackles the challenge of factuality in retrieval-augmented in-context learning by introducing a Hierarchical Graph of Thoughts that dynamically constructs a multi-layer DAG to decompose complex questions. It combines emergent planning, improved self-consistency voting, and a quality-aware passage scoring mechanism that jointly consider thought quality and retrieval relevance. Empirical results on FEVER, Open-SQuAD, and HotPotQA show HGOT outperforms several baselines on factuality (up to 7% EM on FEVER) and matches leading Retrieve-then-Read and DSP approaches across the others, with ablations confirming the value of thought and retrieval quality and identifying robust hyperparameters. The framework offers a practical, scalable approach to reduce hallucinations in LLMs by tightly coupling reasoning structure with retrieval quality, enabling more reliable factuality in real-world tasks.

Abstract

With the widespread adoption of large language models (LLMs) in numerous applications, the challenge of factuality and the propensity for hallucinations has emerged as a significant concern. To address this issue, particularly in retrieval-augmented in-context learning, we introduce the hierarchical graph of thoughts (HGOT), a structured, multi-layered graph approach designed to enhance the retrieval of pertinent passages during in-context learning. The framework utilizes the emergent planning capabilities of LLMs, employing the divide-and-conquer strategy to break down complex queries into manageable sub-queries. It refines self-consistency majority voting for answer selection, which incorporates the recently proposed citation recall and precision metrics to assess the quality of thoughts, linking an answer's credibility intrinsically to the thought's quality. This methodology introduces a weighted system in majority voting, prioritizing answers based on the citation quality of their thoughts. Additionally, we propose a scoring mechanism for evaluating retrieved passages, considering factors such as citation frequency and quality, self-consistency confidence, and the retrieval module's ranking. Experiments indicate that HGOT excels as a versatile approach, outperforming competing models in FEVER by up to $7\%$ and matching leading models such as Retrieve-then-Read in Open-SQuAD, and DSP in HotPotQA, demonstrating its efficacy in enhancing LLMs' factuality.

Figures (3)

• Figure 1: An illustrative example of HGOT in answering a factual question. (The abbreviations employed are as follows: Instr.: Instructions, Q: Question, Ctx.: Context or References, Resp.: ChatGPT's Response, PL: Plan, D: Dependencies, CI: Confidence, Ans.: Answer, Thot.: Thought)
• Figure 2: The sentence length, measured by the number of tokens in a question, from the FEVER, Open-SQuAD, and HotPotQA datasets
• Figure 3: The visualizations of the hyperparameter searches are shown through pairwise relationships, featuring the EM score in the row and hyperparameters $\alpha$, $\beta$, $\gamma$, $w_1$, $w_2$, and $w_3$ in the columns. Each subplot is represented as a line chart, aggregating the data to display the mean (solid blue line) and the $95\%$ confidence interval (light blue area). Additionally, the optimal hyperparameters for attaining the highest EM score are indicated in each subplot.