Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models

Zhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang

TL;DR

This work investigates whether large language models exhibit an evolution from traditional Aristotelian logic to modern Boolean logic in syllogistic reasoning, using existential import as a probing mechanism. A large, carefully constructed dataset of 9600 syllogisms across languages and moods supports a dual-logic evaluation comparing traditional ($Acc_t$) and modern ($Acc_m$) semantics, with metrics for consistency and precision/recall. The study reveals that model size and reinforcement-learning–driven thinking jointly promote a shift toward modern logic, though the transition can be unstable and model-architecture dependent; base models seed the post-training trajectory and can either facilitate or constrain the shift. CoT prompting and distillation show limited impact, while RL-based thinking can match or exceed the modern-logic performance of larger models, highlighting the importance of post-training reasoning policies alongside scaling. Overall, modern logic behavior in LLMs emerges from a combination of base initialization and task-tailored post-training, with notable cross-lingual and architectural variations and several persistent failure modes related to empty minor terms and mood-specific biases.

Abstract

Human logic has gradually shifted from intuition-driven inference to rigorous formal systems. Motivated by recent advances in large language models (LLMs), we explore whether LLMs exhibit a similar evolution in the underlying logical framework. Using existential import as a probe, we for evaluate syllogism under traditional and modern logic. Through extensive experiments of testing SOTA LLMs on a new syllogism dataset, we have some interesting findings: (i) Model size scaling promotes the shift toward modern logic; (ii) Thinking serves as an efficient accelerator beyond parameter scaling; (iii) the Base model plays a crucial role in determining how easily and stably this shift can emerge. Beyond these core factors, we conduct additional experiments for in-depth analysis of properties of current LLMs on syllogistic reasoning.

A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models

TL;DR

This work investigates whether large language models exhibit an evolution from traditional Aristotelian logic to modern Boolean logic in syllogistic reasoning, using existential import as a probing mechanism. A large, carefully constructed dataset of 9600 syllogisms across languages and moods supports a dual-logic evaluation comparing traditional () and modern () semantics, with metrics for consistency and precision/recall. The study reveals that model size and reinforcement-learning–driven thinking jointly promote a shift toward modern logic, though the transition can be unstable and model-architecture dependent; base models seed the post-training trajectory and can either facilitate or constrain the shift. CoT prompting and distillation show limited impact, while RL-based thinking can match or exceed the modern-logic performance of larger models, highlighting the importance of post-training reasoning policies alongside scaling. Overall, modern logic behavior in LLMs emerges from a combination of base initialization and task-tailored post-training, with notable cross-lingual and architectural variations and several persistent failure modes related to empty minor terms and mood-specific biases.

Abstract

Human logic has gradually shifted from intuition-driven inference to rigorous formal systems. Motivated by recent advances in large language models (LLMs), we explore whether LLMs exhibit a similar evolution in the underlying logical framework. Using existential import as a probe, we for evaluate syllogism under traditional and modern logic. Through extensive experiments of testing SOTA LLMs on a new syllogism dataset, we have some interesting findings: (i) Model size scaling promotes the shift toward modern logic; (ii) Thinking serves as an efficient accelerator beyond parameter scaling; (iii) the Base model plays a crucial role in determining how easily and stably this shift can emerge. Beyond these core factors, we conduct additional experiments for in-depth analysis of properties of current LLMs on syllogistic reasoning.
Paper Structure (49 sections, 7 equations, 3 figures, 10 tables)

This paper contains 49 sections, 7 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Dataset Construction
  3. Syllogism and Existential Import
  4. Dataset Construction
  5. Experiment Design
  6. The 15+9 Distinction of Valid Syllogistic Forms
  7. Results and Analysis
  8. Main Results
  9. Scaling Effects of Logical Evolution
  10. Advanced models exhibit modern-logic behavior.
  11. Clear family-specific scaling patterns.
  12. Thinking as an Efficient Driver of the Logic Evolution
  13. Thinking accelerates the logic shift at fixed scale.
  14. Thinking is an efficient alternative to parameter scaling.
  15. CoT Prompting and Distillation are insufficient.
  16. ...and 34 more sections

Figures (3)

  • Figure 1: The illustration of existential import problem and the trace of model logic.
  • Figure 2: Overall performance of auto-regressive models under traditional logic and modern logic. The upper figure shows model performance under the traditional logic criterion, while the lower panel reports performance under the modern logic criterion. Point size is proportional to model scale, and color denotes model family. Qwen-T indicates Qwen Thinking models/mode. For closed-source models, we use a fixed medium point size for visualization only, which does not reflect their true parameter counts. The horizontal dashed line marks the dividing line between the traditional and modern logic.
  • Figure 3: The heatmaps of two types of model logic. (a) and (b) are traditional logic while (c) and (d) are modern logic.