Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy

Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, Rui Yan

TL;DR

DetLR introduces a novel indeterminacy-to-determinacy framework for LLM-based reasoning, segmenting premises into determinate and indeterminate categories and leveraging premise prioritization, three-fold verification, and a reasoning memory to guide iterative deduction. The approach reduces reasoning steps while improving accuracy across multiple logical benchmarks, with strong ablations validating the necessity of each component. Case studies and scalability analyses illustrate robustness to context length and premise count, highlighting improved efficiency over existing multi-step methods. The work advances human-like logical reasoning in LLMs, with practical implications for more reliable automated reasoning systems and future work on implicit-condition extraction.

Abstract

Recent advances in large language models (LLMs) have revolutionized the landscape of reasoning tasks. To enhance the capabilities of LLMs to emulate human reasoning, prior studies have focused on modeling reasoning steps using various thought structures like chains, trees, or graphs. However, LLM-based reasoning still encounters the following challenges: (1) Limited adaptability of preset structures to diverse tasks; (2) Insufficient precision in exploiting known conditions to derive new ones; and (3) Inadequate consideration of historical reasoning experiences for subsequent reasoning steps. To this end, we propose DetermLR, a novel perspective that rethinks the reasoning process as an evolution from indeterminacy to determinacy. First, we categorize known conditions into two types: determinate and indeterminate premises This provides an oveall direction for the reasoning process and guides LLMs in converting indeterminate data into progressively determinate insights. Subsequently, we leverage quantitative measurements to prioritize more relevant premises to explore new insights. Furthermore, we automate the storage and extraction of available premises and reasoning paths with reasoning memory, preserving historical reasoning details for subsequent reasoning steps. Comprehensive experimental results demonstrate that DetermLR surpasses all baselines on various logical reasoning benchmarks: LogiQA, ProofWriter, FOLIO, PrOntoQA, and LogicalDeduction. Compared to previous multi-step reasoning methods, DetermLR achieves higher accuracy with fewer reasoning steps, highlighting its superior efficiency and effectiveness in solving logical reasoning tasks.

DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy

TL;DR

DetLR introduces a novel indeterminacy-to-determinacy framework for LLM-based reasoning, segmenting premises into determinate and indeterminate categories and leveraging premise prioritization, three-fold verification, and a reasoning memory to guide iterative deduction. The approach reduces reasoning steps while improving accuracy across multiple logical benchmarks, with strong ablations validating the necessity of each component. Case studies and scalability analyses illustrate robustness to context length and premise count, highlighting improved efficiency over existing multi-step methods. The work advances human-like logical reasoning in LLMs, with practical implications for more reliable automated reasoning systems and future work on implicit-condition extraction.

Abstract

Recent advances in large language models (LLMs) have revolutionized the landscape of reasoning tasks. To enhance the capabilities of LLMs to emulate human reasoning, prior studies have focused on modeling reasoning steps using various thought structures like chains, trees, or graphs. However, LLM-based reasoning still encounters the following challenges: (1) Limited adaptability of preset structures to diverse tasks; (2) Insufficient precision in exploiting known conditions to derive new ones; and (3) Inadequate consideration of historical reasoning experiences for subsequent reasoning steps. To this end, we propose DetermLR, a novel perspective that rethinks the reasoning process as an evolution from indeterminacy to determinacy. First, we categorize known conditions into two types: determinate and indeterminate premises This provides an oveall direction for the reasoning process and guides LLMs in converting indeterminate data into progressively determinate insights. Subsequently, we leverage quantitative measurements to prioritize more relevant premises to explore new insights. Furthermore, we automate the storage and extraction of available premises and reasoning paths with reasoning memory, preserving historical reasoning details for subsequent reasoning steps. Comprehensive experimental results demonstrate that DetermLR surpasses all baselines on various logical reasoning benchmarks: LogiQA, ProofWriter, FOLIO, PrOntoQA, and LogicalDeduction. Compared to previous multi-step reasoning methods, DetermLR achieves higher accuracy with fewer reasoning steps, highlighting its superior efficiency and effectiveness in solving logical reasoning tasks.
Paper Structure (37 sections, 7 equations, 3 figures, 5 tables)

This paper contains 37 sections, 7 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conventional Logical Reasoning
  4. LLM-based Logical Reasoning
  5. DetermLR
  6. Problem Formulation
  7. Premise Identification
  8. Premise Prioritization and Exploration
  9. Premise prioritization with two-stage scoring.
  10. Premise exploration with three-fold verification.
  11. Reasoning Memorization
  12. Memory storage.
  13. Memory extraction.
  14. Experiments
  15. Experimental Setup
  16. ...and 22 more sections

Figures (3)

  • Figure 1: The overview of DetermLR: (a) premise identification; (b) iterative reasoning process: (b-1) premise prioritization and exploration and (b-2) reasoning memorization. Green elements represent determinate premises, and blue elements represent indeterminate premises. The proportion of blue decreases with the accumulation of green during iterative reasoning.
  • Figure 2: Two cases for contrasting reasoning structure and context complexity. Green dots with $d$ represent determinate premises and blue dots with $i$ represent indeterminate premises.
  • Figure 3: The impact of the number of generated determinate premises.