Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faithful Logical Reasoning via Symbolic Chain-of-Thought

Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, Wynne Hsu

TL;DR

This paper introduces Symbolic Chain-of-Thought (SymbCoT), a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting to improve logical reasoning. It comprises four modules—Translator, Planner, Solver, and Verifier—enabling end-to-end translation, plan-driven reasoning, and retrospective verification, all performed within the LLM without external tools. Evaluations on five standard datasets across First-Order Logic and Constraint Optimization show that SymbCoT yields consistent, substantial gains over vanilla CoT and several state-of-the-art baselines, particularly on more complex tasks, while enhancing faithfulness and explainability through the Verifier. The work demonstrates that combining symbolic representations with natural language, guided by a plan-then-solve approach and rigorous verification, yields more robust and interpretable logical reasoning with LLMs, and it provides open-source code for replication.

Abstract

While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rigid deducing rules. To strengthen the logical reasoning capability of LLMs, we propose a novel Symbolic Chain-of-Thought, namely SymbCoT, a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting. Technically, building upon an LLM, SymbCoT 1) first translates the natural language context into the symbolic format, and then 2) derives a step-by-step plan to solve the problem with symbolic logical rules, 3) followed by a verifier to check the translation and reasoning chain. Via thorough evaluations on 5 standard datasets with both First-Order Logic and Constraint Optimization symbolic expressions, SymbCoT shows striking improvements over the CoT method consistently, meanwhile refreshing the current state-of-the-art performances. We further demonstrate that our system advances in more faithful, flexible, and explainable logical reasoning. To our knowledge, this is the first to combine symbolic expressions and rules into CoT for logical reasoning with LLMs. Code is open at https://github.com/Aiden0526/SymbCoT.

Faithful Logical Reasoning via Symbolic Chain-of-Thought

TL;DR

This paper introduces Symbolic Chain-of-Thought (SymbCoT), a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting to improve logical reasoning. It comprises four modules—Translator, Planner, Solver, and Verifier—enabling end-to-end translation, plan-driven reasoning, and retrospective verification, all performed within the LLM without external tools. Evaluations on five standard datasets across First-Order Logic and Constraint Optimization show that SymbCoT yields consistent, substantial gains over vanilla CoT and several state-of-the-art baselines, particularly on more complex tasks, while enhancing faithfulness and explainability through the Verifier. The work demonstrates that combining symbolic representations with natural language, guided by a plan-then-solve approach and rigorous verification, yields more robust and interpretable logical reasoning with LLMs, and it provides open-source code for replication.

Abstract

While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rigid deducing rules. To strengthen the logical reasoning capability of LLMs, we propose a novel Symbolic Chain-of-Thought, namely SymbCoT, a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting. Technically, building upon an LLM, SymbCoT 1) first translates the natural language context into the symbolic format, and then 2) derives a step-by-step plan to solve the problem with symbolic logical rules, 3) followed by a verifier to check the translation and reasoning chain. Via thorough evaluations on 5 standard datasets with both First-Order Logic and Constraint Optimization symbolic expressions, SymbCoT shows striking improvements over the CoT method consistently, meanwhile refreshing the current state-of-the-art performances. We further demonstrate that our system advances in more faithful, flexible, and explainable logical reasoning. To our knowledge, this is the first to combine symbolic expressions and rules into CoT for logical reasoning with LLMs. Code is open at https://github.com/Aiden0526/SymbCoT.
Paper Structure (75 sections, 2 equations, 10 figures, 4 tables)

This paper contains 75 sections, 2 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. SymbCoT for Symbolic Reasoning
  4. Task Definition
  5. Modules
  6. Translator
  7. Planner
  8. Solver
  9. Verifier
  10. Reasoning Steps
  11. Step 1: Translating natural language context into symbolic.
  12. Step 2: Deriving plan based on the natural and symbolic context.
  13. Step 3: Solving the problem given context and plan.
  14. Step4: Verifying the translation and solving process.
  15. Experiments
  16. ...and 60 more sections

Figures (10)

  • Figure 1: An illustrative example of logical reasoning via Chain-of-Thought and our proposed Symbolic CoT (SymbCoT).
  • Figure 2: Overview of the workflow in our proposed symbolic CoT framework.
  • Figure 3: Ablation study. Since the Solver is dependent on the Planner, they have to be ablated simultaneously.
  • Figure 4: The effect of reasoning depth with GPT-4 on ProofWriter. The red dual-head arrow indicates our improvements over vanilla CoT.
  • Figure 5: Execution rate between Logic-LM and Ours.
  • ...and 5 more figures