Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs

Kangsheng Wang, Xiao Zhang, Hao Liu, Songde Han, Huimin Ma, Tianyu Hu

TL;DR

By integrating CRE and DES (CreDes), the model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT).

Abstract

Large language models (LLMs) have demonstrated limitations in handling combinatorial optimization problems involving long-range reasoning, partially due to causal hallucinations and huge search space. As for causal hallucinations, i.e., the inconsistency between reasoning and corresponding state transition, this paper introduces the Causal Relationship Enhancement (CRE) mechanism combining cause-effect interventions and the Individual Treatment Effect (ITE) to guarantee the solid causal rightness between each step of reasoning and state transition. As for the long causal range and huge search space limiting the performances of existing models featuring single-direction search, a Dual-End Searching (DES) approach is proposed to seek solutions by simultaneously starting from both the initial and goal states on the causal probability tree. By integrating CRE and DES (CreDes), our model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT). Experiments demonstrate that CreDes significantly outperforms existing State-Of-The-Art (SOTA) solutions in long-range reasoning tasks in terms of both accuracy and time efficiency.

CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs

TL;DR

By integrating CRE and DES (CreDes), the model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT).

Abstract

Large language models (LLMs) have demonstrated limitations in handling combinatorial optimization problems involving long-range reasoning, partially due to causal hallucinations and huge search space. As for causal hallucinations, i.e., the inconsistency between reasoning and corresponding state transition, this paper introduces the Causal Relationship Enhancement (CRE) mechanism combining cause-effect interventions and the Individual Treatment Effect (ITE) to guarantee the solid causal rightness between each step of reasoning and state transition. As for the long causal range and huge search space limiting the performances of existing models featuring single-direction search, a Dual-End Searching (DES) approach is proposed to seek solutions by simultaneously starting from both the initial and goal states on the causal probability tree. By integrating CRE and DES (CreDes), our model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT). Experiments demonstrate that CreDes significantly outperforms existing State-Of-The-Art (SOTA) solutions in long-range reasoning tasks in terms of both accuracy and time efficiency.
Paper Structure (25 sections, 10 equations, 5 figures, 5 tables, 2 algorithms)

This paper contains 25 sections, 10 equations, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Definition
  5. Causal Significance and Consistancy
  6. Causal Relationship Enhancement (CRE)
  7. Causal Probability Trees with Dual End Searching (DES)
  8. Experiment
  9. Setup
  10. Dataset and Basemodel
  11. Benchmark
  12. Results
  13. Discussion
  14. Limitation
  15. Conclusion
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Integrating Causal Relationship Enhancement (CRE) and Dual-End Searching (DES).
  • Figure 2: Schematic illustration of Causal Relationship Enhancement(CRE).
  • Figure 3: Exact Frequency and Cumulative Histogram for Scenarios A, B, and C. For the convenience of image drawing, the coordinate axes of this image have been scaled to a certain degree and do not represent the actual values.
  • Figure 4: Sketch of the DES (in 9-steps)
  • Figure 5: Improvement in reasoning speed for long-range tasks (based on a single A100 GPU).