Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus

Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, Yasuhiro Sogawa

TL;DR

The paper tackles the limited reasoning capabilities of large language models by introducing Additional Logic Training (ALT) using a systematically designed synthetic logic corpus. It establishes design principles grounded in symbolic logic and empirical findings, then builds the FLD_{\times \mathbbm{2}} corpus to train LLMs on multi-step deductions involving unknown facts, diverse rules, and varied linguistic expressions. Empirical results show substantial gains in logical reasoning (up to ~30 points), with notable improvements in math, coding, and NLI, and ablations confirm the necessity of each design principle and forgetting-prevention via Recall Adam. The work demonstrates that reasoning-trained models can generalize beyond the synthetic tasks and proposes releasing the corpus, code, and trained models to support reproducibility and broader impact. Overall, ALT offers a principled path to more versatile AI that fuses knowledge with robust deductive reasoning.

Abstract

Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning. To address this, we propose $\textbf{Additional Logic Training (ALT)}$, which aims to enhance LLMs' reasoning capabilities by program-generated logical reasoning samples. We first establish principles for designing high-quality samples by integrating symbolic logic theory and previous empirical insights. Then, based on these principles, we construct a synthetic corpus named $\textbf{Formal Logic Deduction Diverse}$ ($\textbf{FLD}$$_{\times 2}$), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors. Finally, we empirically show that ALT on FLD$_{\times2}$ substantially enhances the reasoning capabilities of state-of-the-art LLMs, including LLaMA-3.1-70B. Improvements include gains of up to 30 points on logical reasoning benchmarks, up to 10 points on math and coding benchmarks, and 5 points on the benchmark suite BBH.

Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus

TL;DR

The paper tackles the limited reasoning capabilities of large language models by introducing Additional Logic Training (ALT) using a systematically designed synthetic logic corpus. It establishes design principles grounded in symbolic logic and empirical findings, then builds the FLD_{\times \mathbbm{2}} corpus to train LLMs on multi-step deductions involving unknown facts, diverse rules, and varied linguistic expressions. Empirical results show substantial gains in logical reasoning (up to ~30 points), with notable improvements in math, coding, and NLI, and ablations confirm the necessity of each design principle and forgetting-prevention via Recall Adam. The work demonstrates that reasoning-trained models can generalize beyond the synthetic tasks and proposes releasing the corpus, code, and trained models to support reproducibility and broader impact. Overall, ALT offers a principled path to more versatile AI that fuses knowledge with robust deductive reasoning.

Abstract

Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning. To address this, we propose , which aims to enhance LLMs' reasoning capabilities by program-generated logical reasoning samples. We first establish principles for designing high-quality samples by integrating symbolic logic theory and previous empirical insights. Then, based on these principles, we construct a synthetic corpus named (), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors. Finally, we empirically show that ALT on FLD substantially enhances the reasoning capabilities of state-of-the-art LLMs, including LLaMA-3.1-70B. Improvements include gains of up to 30 points on logical reasoning benchmarks, up to 10 points on math and coding benchmarks, and 5 points on the benchmark suite BBH.

Paper Structure

This paper contains 34 sections, 1 theorem, 7 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. How Should Synthetic Logic Samples Be Designed?
  3. Teaching Reasoning with Unknown Facts
  4. Teaching Illogical Reasoning
  5. Teaching Diverse Reasoning Rules
  6. Teaching Diverse Linguistic Expressions that Represent Logical Statements
  7. Creating a Synthetic Corpus based on Design Principles
  8. Experimental Setup
  9. Can Additional Logic Training Enhance LLMs' Capabilities?
  10. What Capabilities Can Additional Logic Training Enhance and Why?
  11. Logical Reasoning Tasks
  12. Math and Coding Tasks
  13. NLI Tasks
  14. Other Tasks
  15. Conclusion
  16. ...and 19 more sections

Key Result

Theorem 2.1

Any valid deduction rule can be expressed by multistep deductive reasoning constructed from the axioms.

Figures (3)

  • Figure 1: The performance gains to LLaMA-3.1-70B by Additional Logic Training (ALT) on the proposed synthetic corpus, FLD$_{\times \mathbbm{2}}$ (Formal Logic Deduction Diverse). Each benchmark set, such as "Logic" and "Math", comprises various benchmarks in that domain. \ref{['tb:performance_aggregated', 'tb:performance_details']} shows the details.
  • Figure 2: Our proposed Additional Logic Training (ALT) aims to enhance LLMs' reasoning capabilities through training on many synthetically generated logical reasoning samples. Our sample generator (left) first generates a sample of multi-step deductive reasoning and then converts it into a deduction sample written in English (right). LLMs must generate logical steps to derive a given hypothesis from provided facts. The sample generator adheres to theoretically and empirically grounded design principles discussed in \ref{['sec:design_principles']}. Refer to \ref{['appendix:fig:deduction_example']} for a real sample.
  • Figure D.3: A real deduction sample included in Formal Logic Deduction Diverse. Facts and hypothesis are given to LLMs, then the LLMs are required to generate logical steps to (dis-)prove the hypothesis based on the facts, and an answer label (see \ref{['appendix:sec:PLD_statistics']}).

Theorems & Definitions (1)

  • Theorem 2.1: Completeness of First-Order Predicate Logic godel1930uber