Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning

Pengfei He, Zitao Li, Yue Xing, Yaling Li, Jiliang Tang, Bolin Ding

TL;DR

A novel structure-oriented analysis method is introduced to help LLMs better understand the question and guide the problem-solving process of LLMs and a multi-agent reasoning system is proposed that can better enforce the reasoning process following the authors' structure-oriented analysis by refinement techniques and is equipped with external knowledge retrieval capability to reduce factual errors.

Abstract

Zero-shot reasoning methods with Large Language Models (LLMs) offer significant advantages including great generalization to novel tasks and reduced dependency on human-crafted examples. However, the current zero-shot methods still have limitations in complex tasks, e.g., answering questions that require multi-step reasoning. In this paper, we address this limitation by introducing a novel structure-oriented analysis method to help LLMs better understand the question and guide the problem-solving process of LLMs. We first demonstrate how the existing reasoning strategies, Chain-of-Thought and ReAct, can benefit from our structure-oriented analysis. In addition to empirical investigations, we leverage the probabilistic graphical model to theoretically explain why our structure-oriented analysis can improve the LLM reasoning process. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA), that can better enforce the reasoning process following our structure-oriented analysis by refinement techniques and is equipped with external knowledge retrieval capability to reduce factual errors. Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods. Finally, the system not only improves reasoning accuracy in complex tasks but also demonstrates robustness against potential attacks that corrupt the reasoning process.

Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning

TL;DR

A novel structure-oriented analysis method is introduced to help LLMs better understand the question and guide the problem-solving process of LLMs and a multi-agent reasoning system is proposed that can better enforce the reasoning process following the authors' structure-oriented analysis by refinement techniques and is equipped with external knowledge retrieval capability to reduce factual errors.

Abstract

Zero-shot reasoning methods with Large Language Models (LLMs) offer significant advantages including great generalization to novel tasks and reduced dependency on human-crafted examples. However, the current zero-shot methods still have limitations in complex tasks, e.g., answering questions that require multi-step reasoning. In this paper, we address this limitation by introducing a novel structure-oriented analysis method to help LLMs better understand the question and guide the problem-solving process of LLMs. We first demonstrate how the existing reasoning strategies, Chain-of-Thought and ReAct, can benefit from our structure-oriented analysis. In addition to empirical investigations, we leverage the probabilistic graphical model to theoretically explain why our structure-oriented analysis can improve the LLM reasoning process. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA), that can better enforce the reasoning process following our structure-oriented analysis by refinement techniques and is equipped with external knowledge retrieval capability to reduce factual errors. Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods. Finally, the system not only improves reasoning accuracy in complex tasks but also demonstrates robustness against potential attacks that corrupt the reasoning process.

Paper Structure

This paper contains 26 sections, 3 theorems, 21 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. structure-oriented analysis
  4. Empirical findings
  5. Theoretical analysis
  6. Autonomous reasoning system
  7. System Design
  8. Reasoning process
  9. Experiments
  10. Experiment setting
  11. Main performance
  12. Effect of structure-oriented analysis
  13. Effect of key agents
  14. Evaluation of robustness
  15. Conclusion
  16. ...and 11 more sections

Key Result

Lemma 3.2

Let $\Gamma_A(x_0,\cdot,\mathcal{M})$ denote the set of explored paths given $A$. Under Assumption assumption:gamma, assume that $A\subseteq \gamma^*$, then the following results in $\theta_T$ (with the corresponding index $T$) and $\gamma$ hold: (1) When $|A|=1$, i.e. $A=\{s^A\}$ for some $s^A\in\g

Figures (5)

  • Figure 1: An illustration of the structure-oriented analysis
  • Figure 2: Reasoning accuracy with/without the structure-oriented analysis. The methods with suffixes $+$ are the backbone methods ({CoT, ReAct} $\times$ {0-shot, 6-shot}) with structure-oriented analysis added.
  • Figure 3: An illustrative example of the PGM generation model. This graph is a part of the underlying PGM where $\theta_i$s are hidden variables and $x_i$s are observed variables. The red circle is an example of the strong connection between $\theta_i$s and $x_i$s in the pre-training.
  • Figure 4: An overview of the Structure-oriented Autonomous Reasoning Agents.
  • Figure 5: Ablation study on agents. Refinement Agent and Retrieval Agent are removed and reasoning performance is tested respectively.

Theorems & Definitions (8)

  • Lemma 3.2
  • Theorem 3.3
  • Remark 3.4: Multiple correct paths
  • Remark 3.5: Error when the exploration is not guaranteed to find $\theta_s$ for some $s\in A$
  • proof : Proof of Lemma \ref{['lem:main']}
  • proof : Proof of Theorem \ref{['them:main']}, 0-1 error
  • Lemma A.1: Decomposition of probability error.
  • proof : Proof of Theorem \ref{['them:main']}, probability error