ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems

TL;DR

ChaosBench-Logic introduces a rigorous, ontology-grounded benchmark to diagnose logical and symbolic reasoning in LLMs within chaotic dynamical systems. By anchoring all items to a global first-order logic axiom system $\Phi$ and a ground-truth closure $\mathrm{Cl}_\Phi$, the benchmark reveals that state-of-the-art models achieve high per-item accuracy ($\approx 91{-}94\%$) but struggle with compositional reasoning and maintaining global coherence across dialogue turns. Seven task families, 621 questions across 30 systems, and 11 predicates enable fine-grained analysis of local versus global reasoning, bias susceptibility, and dialogue consistency. The findings underscore a substantial gap between item-level success and coherent, constraint-satisfying reasoning in scientific domains, motivating neuro-symbolic and tool-augmented approaches to improve long-horizon, domain-grounded reasoning. ChaosBench-Logic thus provides a scalable, auditable platform to drive advances in globally consistent scientific reasoning for LLM-based systems.

Abstract

Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.