Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Tasks

Wentao Wan, Zhuojie Yang, Yongcan Chen, Chenglin Luo, Ruilin Wang, Kehao Cai, Nan Kang, Liang Lin, Keze Wang

TL;DR

This work tackles the challenge of rigorous deductive reasoning in large language models by introducing SR-FoT, a five-stage syllogistic framework that guides LLMs to autonomously generate a major premise and a minor premise before performing formal syllogistic deduction. By restricting information flow at each stage, SR-FoT aims to reduce reasoning illusions common in Chain-of-Thought while leveraging built-in knowledge for complex tasks. Comprehensive experiments across ScienceQA, StrategyQA, and BoolQ show that SR-FoT and its self-consistent variant outperform CoT and related baselines across GPT-3.5-turbo, DeepSeek-v2, and Qwen1.5-32B-Chat, with notable gains on open models. The results also include ablation and rigor analyses indicating that each stage and the visibility of information are critical to achieving rigorous, reliable reasoning.

Abstract

Deductive reasoning is a crucial logical capability that assists us in solving complex problems based on existing knowledge. Although augmented by Chain-of-Thought prompts, Large Language Models (LLMs) might not follow the correct reasoning paths. Enhancing the deductive reasoning abilities of LLMs, and leveraging their extensive built-in knowledge for various reasoning tasks, remains an open question. Attempting to mimic the human deductive reasoning paradigm, we propose a multi-stage Syllogistic-Reasoning Framework of Thought (SR-FoT) that enables LLMs to perform syllogistic deductive reasoning to handle complex knowledge-based reasoning tasks. Our SR-FoT begins by interpreting the question and then uses the interpretation and the original question to propose a suitable major premise. It proceeds by generating and answering minor premise questions in two stages to match the minor premises. Finally, it guides LLMs to use the previously generated major and minor premises to perform syllogistic deductive reasoning to derive the answer to the original question. Extensive and thorough experiments on knowledge-based reasoning tasks have demonstrated the effectiveness and advantages of our SR-FoT.

SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Tasks

TL;DR

This work tackles the challenge of rigorous deductive reasoning in large language models by introducing SR-FoT, a five-stage syllogistic framework that guides LLMs to autonomously generate a major premise and a minor premise before performing formal syllogistic deduction. By restricting information flow at each stage, SR-FoT aims to reduce reasoning illusions common in Chain-of-Thought while leveraging built-in knowledge for complex tasks. Comprehensive experiments across ScienceQA, StrategyQA, and BoolQ show that SR-FoT and its self-consistent variant outperform CoT and related baselines across GPT-3.5-turbo, DeepSeek-v2, and Qwen1.5-32B-Chat, with notable gains on open models. The results also include ablation and rigor analyses indicating that each stage and the visibility of information are critical to achieving rigorous, reliable reasoning.

Abstract

Deductive reasoning is a crucial logical capability that assists us in solving complex problems based on existing knowledge. Although augmented by Chain-of-Thought prompts, Large Language Models (LLMs) might not follow the correct reasoning paths. Enhancing the deductive reasoning abilities of LLMs, and leveraging their extensive built-in knowledge for various reasoning tasks, remains an open question. Attempting to mimic the human deductive reasoning paradigm, we propose a multi-stage Syllogistic-Reasoning Framework of Thought (SR-FoT) that enables LLMs to perform syllogistic deductive reasoning to handle complex knowledge-based reasoning tasks. Our SR-FoT begins by interpreting the question and then uses the interpretation and the original question to propose a suitable major premise. It proceeds by generating and answering minor premise questions in two stages to match the minor premises. Finally, it guides LLMs to use the previously generated major and minor premises to perform syllogistic deductive reasoning to derive the answer to the original question. Extensive and thorough experiments on knowledge-based reasoning tasks have demonstrated the effectiveness and advantages of our SR-FoT.
Paper Structure (22 sections, 4 figures, 6 tables)

This paper contains 22 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chain-of-Thought
  4. Logical Reasoning Ability of LLMs
  5. Methodology
  6. Background: Syllogism
  7. Procedure of Our SR-FoT
  8. Experiments
  9. Experiment Setup
  10. Datasets.
  11. Experimental Setting.
  12. Experimental Results and Analyses
  13. Performance on ScienceQA.
  14. Performance on StrategyQA and BoolQ.
  15. Ablation Study
  16. ...and 7 more sections

Figures (4)

  • Figure 1: A syllogism example.
  • Figure 2: Procedure of our SR-FoT. Questionori: Original Question, Context: Context provided for Original Question, Answerori: Answer for Original Question, QuestionmP: Question for Minor Premise, PropmtCoT: Guide Prompt for CoT, PropmtQE: Guide Prompt for Question Explanation, PropmtMP: Guide Prompt for Major Premise Production, PropmtQmP: Guide Prompt for Posing the Minor Premise Question, PromptmP: Guide Prompt for Minor Premise Production, PromptSR: Guide Prompt for Final Syllogistic Reasoning and so on.
  • Figure 3: Prompts for each stage of our SR-FoT.
  • Figure 4: A case of using CoT and SR-FoT to answer a question in the ScienceQA dataset respectively. The highlighted red parts indicate the incorrect or misleading content, while the highlighted green parts indicate the content that helps correct reasoning.