Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

TL;DR

This work tackles the persistent challenge of unfaithful and incomplete reasoning in large language models by introducing Logic-of-Thought (LoT), a zero-shot prompting framework that uses propositional logic to generate expanded logical descriptions and injects them into prompts without losing the original context. LoT operates in three phases—Logic Extraction, Logic Extension, and Logic Translation—where LLMs extract propositions and relations, a Python module expands these expressions via logical laws, and the results are translated back into natural language as augmentation to the input. The approach is orthogonal to existing prompting methods and demonstrates substantial gains across five logical datasets (ReClor, LogiQA, RuleTaker, ProofWriter, FOLIO) when combined with CoT, SC, CoT-SC, or ToT, including notable improvements such as +4.35% on ReClor for CoT and +8% for ToT on ProofWriter. Across comparisons with neuro-symbolic baselines (SatLM, LINC), LoT generally yields superior accuracy and robustness, though certain extraction biases can limit performance in some configurations. Limitations include a restricted set of connectives and potential extraction errors, pointing to future work on expanding logical primitives and mitigating hallucinations, while still offering a practical, integrable enhancement to existing reasoning prompts.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information descriptions and utilizes them as an additional augmentation to original contexts, thereby ensuring information completeness and enhancing logical reasoning ability. LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%, improves Chain-of-Thought with Self-Consistency's performance on the RuleTaker dataset by +3.52%, and boosts performance of Tree-of-Thoughts on the ProofWriter dataset by +8%.

Figures (5)

• Figure 1: Comparison between LINC and LoT. The left part involves the workflow of LINC, which converts problems into logical expressions and then derives results using Prover9. However, LINC loses fact information Person(Harry) and Book(Walden), leading to errors. On the right side, LoT generates supplementary logical information "If a person reads a book, that person become smarter" that is seamlessly integrated into the original prompt, thereby enhancing the capability of LLMs to produce accurate results.
• Figure 2: The framework of LoT consisting of three phases. On the left side of the diagram is the Logic Extraction phase, where we employ LLMs to extract propositions and logical relations. In the middle is the Logic Extension phase, where we apply logical reasoning laws to derive logical expressions. On the right side is the Logic Translation phase, where we utilize LLMs to translate logical expressions into their natural language descriptions.
• Figure 5: Comparison between ToT and LoT+ToT in the ProofWriter dataset. We conducted this experiments based on zhang2024cumulative.
• Figure 6: Case study of state exploration in ToT and LoT+ToT.
• Figure 7: Errors in Logic Extraction using LoT.