Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

Michael K. Chen, Xikun Zhang, Dacheng Tao

TL;DR

JustLogic tackles evaluating deductive reasoning in LLMs by addressing limitations of existing benchmarks. It presents a synthetic, code-generated dataset that achieves high natural-language and argument complexity while ensuring prior-knowledge independence and enabling in-depth error analysis. Experiments show SOTA reasoning LLMs reach around the human average but lag behind the human ceiling, while non-reasoning models still underperform the average human. The suite and data are open-source, offering a scalable, future-proof benchmark for advancing deductive reasoning in LLMs.

Abstract

Logical reasoning is a critical component of Large Language Models (LLMs), and substantial research efforts in recent years have aimed to enhance their deductive reasoning capabilities. However, existing deductive reasoning benchmarks, which are crucial for evaluating and advancing LLMs, are inadequate due to their lack of task complexity, presence of prior knowledge as a confounder, and superficial error analysis. To address these deficiencies, we introduce JustLogic, a synthetically generated deductive reasoning benchmark designed for rigorous evaluation of LLMs. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy. Our experimental results on JustLogic reveal that (i) state-of-the-art (SOTA) reasoning LLMs perform on par or better than the human average but significantly worse than the human ceiling, and (ii) SOTA non-reasoning models still underperform the human average. All code and data are available at https://github.com/michaelchen-lab/JustLogic

JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

TL;DR

JustLogic tackles evaluating deductive reasoning in LLMs by addressing limitations of existing benchmarks. It presents a synthetic, code-generated dataset that achieves high natural-language and argument complexity while ensuring prior-knowledge independence and enabling in-depth error analysis. Experiments show SOTA reasoning LLMs reach around the human average but lag behind the human ceiling, while non-reasoning models still underperform the average human. The suite and data are open-source, offering a scalable, future-proof benchmark for advancing deductive reasoning in LLMs.

Abstract

Logical reasoning is a critical component of Large Language Models (LLMs), and substantial research efforts in recent years have aimed to enhance their deductive reasoning capabilities. However, existing deductive reasoning benchmarks, which are crucial for evaluating and advancing LLMs, are inadequate due to their lack of task complexity, presence of prior knowledge as a confounder, and superficial error analysis. To address these deficiencies, we introduce JustLogic, a synthetically generated deductive reasoning benchmark designed for rigorous evaluation of LLMs. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy. Our experimental results on JustLogic reveal that (i) state-of-the-art (SOTA) reasoning LLMs perform on par or better than the human average but significantly worse than the human ceiling, and (ii) SOTA non-reasoning models still underperform the human average. All code and data are available at https://github.com/michaelchen-lab/JustLogic
Paper Structure (34 sections, 8 figures, 10 tables, 1 algorithm)

This paper contains 34 sections, 8 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Existing reasoning datasets for Large Language Models
  4. Reasoning in Large Language Models
  5. Dataset Construction
  6. Step 1: Generate argument structure
  7. Step 2: Adding natural language
  8. Step 3: Generate query statement
  9. Dataset Complexity
  10. Future-proofing JustLogic
  11. Experimental Setup
  12. Prior Knowledge Independence Test
  13. Evaluation of LLMs' Deductive Reasoning
  14. Results
  15. Prior Knowledge Independence Test
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Example of a question adapted using JustLogic's dataset construction algorithm. Formal notations are included for illustrative purposes and are not provided to models.
  • Figure 2: A step-by-step example of how an instance with a reasoning depth of 2 is constructed.
  • Figure 3: How argument form and reasoning depth affects accuracy for various models.
  • Figure 4: Example of a prior knowledge independence test prompt.
  • Figure 5: How factual accuracy of conclusions affects model accuracy.
  • ...and 3 more figures