Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoReason: Automatic Few-Shot Reasoning Decomposition

Arda Sevinc, Abdurrahman Gumus

TL;DR

AutoReason proposes automatic generation of Chain-of-Thought rationales to decompose implicit queries into explicit sub-questions, enabling a zero-shot prompt to behave like per-query few-shot reasoning traces. The system uses a two-stage process where a strong LLM (GPT-4) generates rationales that guide a weaker LLM (GPT-3.5-Turbo) to produce final answers, improving reasoning on StrategyQA and showing mixed results on HotpotQA. Results indicate substantial gains on StrategyQA, demonstrating the potential for per-query rationale generation to enhance interpretability and accuracy, while highlighting task-dependent limitations. The work contributes a scalable methodology for interpretability-driven prompting and provides release of the code to foster further research.

Abstract

Chain of Thought (CoT) was introduced in recent research as a method for improving step-by-step reasoning in Large Language Models. However, CoT has limited applications such as its need for hand-crafted few-shot exemplar prompts and no capability to adjust itself to different queries. In this work, we propose a system to automatically generate rationales using CoT. Our method improves multi-step implicit reasoning capabilities by decomposing the implicit query into several explicit questions. This provides interpretability for the model, improving reasoning in weaker LLMs. We test our approach with two Q\&A datasets: StrategyQA and HotpotQA. We show an increase in accuracy with both, especially on StrategyQA. To facilitate further research in this field, the complete source code for this study has been made publicly available on GitHub: https://github.com/miralab-ai/autoreason.

AutoReason: Automatic Few-Shot Reasoning Decomposition

TL;DR

AutoReason proposes automatic generation of Chain-of-Thought rationales to decompose implicit queries into explicit sub-questions, enabling a zero-shot prompt to behave like per-query few-shot reasoning traces. The system uses a two-stage process where a strong LLM (GPT-4) generates rationales that guide a weaker LLM (GPT-3.5-Turbo) to produce final answers, improving reasoning on StrategyQA and showing mixed results on HotpotQA. Results indicate substantial gains on StrategyQA, demonstrating the potential for per-query rationale generation to enhance interpretability and accuracy, while highlighting task-dependent limitations. The work contributes a scalable methodology for interpretability-driven prompting and provides release of the code to foster further research.

Abstract

Chain of Thought (CoT) was introduced in recent research as a method for improving step-by-step reasoning in Large Language Models. However, CoT has limited applications such as its need for hand-crafted few-shot exemplar prompts and no capability to adjust itself to different queries. In this work, we propose a system to automatically generate rationales using CoT. Our method improves multi-step implicit reasoning capabilities by decomposing the implicit query into several explicit questions. This provides interpretability for the model, improving reasoning in weaker LLMs. We test our approach with two Q\&A datasets: StrategyQA and HotpotQA. We show an increase in accuracy with both, especially on StrategyQA. To facilitate further research in this field, the complete source code for this study has been made publicly available on GitHub: https://github.com/miralab-ai/autoreason.

Paper Structure

This paper contains 21 sections, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. AutoReason
  5. Testing
  6. Results and Discussion
  7. Results
  8. Discussion
  9. Performance Across Datasets
  10. Model Behavior and Regression
  11. Implications for AGI and Complex Reasoning
  12. Ethical Considerations and Societal Impact
  13. Limitations and Future Work
  14. Conclusion
  15. Appendix
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Reasoning methods based on Chain-of-Though (CoT).
  • Figure 2: Few-Shot CoT and Zero-Shot CoT compared with direct Zero-Shot reasoning side by side.
  • Figure 3: Block diagram illustrating the logical flow of prompt transformation of the AutoReason framework.
  • Figure 4: HotpotQA Example Data.
  • Figure 5: StrategyQA implicit reasoning.
  • ...and 1 more figures