From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problems

TL;DR

The paper investigates whether large language models can detect logical inconsistencies in faulty math problems rather than simply computing answers. It introduces FaultyMath, a balanced dataset of $363$ genuinely faulty problems drawn from $635$ candidates, and analyzes LLM performance across detection, hint responsiveness, and explanation quality using a three-part curation pipeline and a robust auto-evaluation framework. Results show that even top models struggle to identify faults without hints, exhibit bias when hints are misleading, and only a subset provide consistently reliable explanations, highlighting a gap between mathematical proficiency and logical reasoning. The work argues for advancing from Blind Solver to Logical Thinker in LLMs and positions FaultyMath as a benchmark to guide future research and prompt-design improvements for more reliable mathematical reasoning.

Abstract

Consider the math problem: "Lily received 3 cookies from her best friend yesterday and ate 5 for breakfast. Today, her friend gave her 3 more cookies. How many cookies does Lily have now?" Many large language models (LLMs) in previous research approach this problem by calculating the answer "1" using the equation "3 - 5 + 3." However, from a human perspective, we recognize the inherent flaw in this problem: Lily cannot eat 5 cookies if she initially only had 3. This discrepancy prompts a key question: Are current LLMs merely Blind Solver that apply mathematical operations without deeper reasoning, or can they function as Logical Thinker capable of identifying logical inconsistencies? To explore this question, we propose a benchmark dataset, FaultyMath, which includes faulty math problems of rich diversity: i) multiple mathematical categories, e.g., algebra, geometry, number theory, etc., ii) varying levels of difficulty, and iii) different origins of faultiness -- ranging from violations of common sense and ambiguous statements to mathematical contradictions and more. We evaluate a broad spectrum of LLMs, including open-source, closed-source, and math-specialized models, using FaultyMath across three dimensions: (i) How accurately can the models detect faulty math problems without being explicitly prompted to do so? (ii) When provided with hints -- either correct or misleading -- about the validity of the problems, to what extent do LLMs adapt to become reliable Logical Thinker? (iii) How trustworthy are the explanations generated by LLMs when they recognize a math problem as flawed? Through extensive experimentation and detailed analysis, our results demonstrate that existing LLMs largely function as Blind Solver and fall short of the reasoning capabilities required to perform as Logical Thinker.

Figures (5)

• Figure 1: A faulty math problem and answers by popular LLMs.
• Figure 2: FaultyMath curation process. Three stages: i) GPT-4 converts valid math problems into faulty ones; ii) GPT-4 self-verifies; iii) human verification.
• Figure 3: Distributions of the dataset by difficulty level and math category (left), and by the origin of falsehood (right).
• Figure 4: Accuracy heatmaps for different difficulty levels and categories. Top-left: Average accuracy of top 3 LLMs. Top-right: Gemini-1.5-Pro (Rank 1). Bottom-left: GPT-4 (Rank 2). Bottom-right: Qwen1.5-72B (Rank 3).
• Figure 5: Accuracy of the top three LLMs across different origins of falsehood.