Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering

TL;DR

Self-Critique Guided Iterative Reasoning (SiGIR) addresses knowledge-intensive multi-hop QA by integrating iterative question decomposition with self-evaluated feedback to guide reasoning trajectories. It trains a reasoning model and a critic to produce self-criticized iterative steps and employs a reward-guided beam search during inference to select promising trajectories. Across HotpotQA, 2WikiMQA, and MuSiQue, SiGIR achieves state-of-the-art performance with an average improvement of 8.6% over prior methods, supported by thorough ablations highlighting the roles of self-critique, reward granularity, and hybrid retrieval. The work provides practical insights into data synthesis, training dynamics, and efficient inference for complex QA tasks, with code and models publicly available.

Abstract

Although large language models (LLMs) have demonstrated remarkable reasoning capabilities, they still face challenges in knowledge-intensive multi-hop reasoning. Recent work explores iterative retrieval to address complex problems. However, the lack of intermediate guidance often results in inaccurate retrieval and flawed intermediate reasoning, leading to incorrect reasoning. To address these, we propose Self-Critique Guided Iterative Reasoning (SiGIR), which uses self-critique feedback to guide the iterative reasoning process. Specifically, through end-to-end training, we enable the model to iteratively address complex problems via question decomposition. Additionally, the model is able to self-evaluate its intermediate reasoning steps. During iterative reasoning, the model engages in branching exploration and employs self-evaluation to guide the selection of promising reasoning trajectories. Extensive experiments on three multi-hop reasoning datasets demonstrate the effectiveness of our proposed method, surpassing the previous SOTA by $8.6\%$. Furthermore, our thorough analysis offers insights for future research. Our code, data, and models are available at Github: https://github.com/zchuz/SiGIR-MHQA.

Figures (8)

• Figure 1: A comparison of iterative retrieval and self-critique guided reasoning (ours) in MHQA. Our method incorporates self-critique during iterative reasoning and conduct searches based on the received feedback.
• Figure 2: Overview of SiGIR. (I) Training recipe to get the iterative reasoner with self-critique, $R_{sc}$. (II) Reasoning characteristic of SC-Reasoner, which includes question decomposition, trigger retrieval, retrieval-augmented reasoning, self-evaluation, and question reduction. (III) SC-Reasoner engages in exploration during the iterative reasoning process and conducts reward-guided searches to select promising reasoning trajectories.
• Figure 3: The impact of increasing training data and self-improvement on performance. As data scales, the performance consistently improves. Starting with 40% of the data, two self-improvement iterations boost performance to nearly 80%, with only 2% gap with full sft.
• Figure 4: Relationship between performance and the average number of iterations to solve a multi-hop question. Upper-left represents efficient reasoning. Our method requires fewer iterations to solve one question with higher F1, demonstrating higher reasoning efficiency.
• Figure 5: The fewshot prompts for iterative reasoning rationales synthesis.