ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models

TL;DR

ReliableMath introduces a principled benchmark to evaluate LLM reliability on mathematical reasoning by balancing solvable and unsolvable problems. It constructs a scalable unsolvable data pipeline that rewrites solvable problems, verifies unsolvability, and validates with human checks, yielding a dataset $ abla_r$ containing $ abla_a$ and $ abla_u$. Through extensive experiments on reasoning and instruction LLMs, the study reveals that reliable prompts stabilize performance on solvable tasks and substantially improve reliability on unsolvable ones, while small LLMs remain less responsive. To address this, the authors propose an alignment framework combining supervised fine-tuning (SFT) and direct preference optimization (DPO) to teach small models to identify solvability and refuse when appropriate, showing gains on both in-domain math and out-of-domain knowledge QA. Overall, ReliableMath provides a rigorous evaluation framework, a scalable data-generation workflow, and a practical alignment strategy to advance reliable mathematical reasoning in LLMs.

Abstract

Although demonstrating remarkable performance on reasoning tasks, Large Language Models (LLMs) still tend to fabricate unreliable responses when confronted with problems that are unsolvable or beyond their capability, severely undermining the reliability. Prior studies of LLM reliability have primarily focused on knowledge tasks to identify unanswerable questions, while mathematical reasoning tasks have remained unexplored due to the dearth of unsolvable math problems. To systematically investigate LLM reliability in mathematical reasoning tasks, we formulate the reliability evaluation for both solvable and unsolvable problems. We then develop a ReliableMath dataset which incorporates open-source solvable problems and high-quality unsolvable problems synthesized by our proposed construction workflow with human evaluations. Experiments are conducted on various LLMs with several key findings uncovered. LLMs fail to directly identify unsolvable problems and always generate fabricated responses. When instructing LLMs to indicate unsolvability using a reliable prompt, the reliability of larger-sized LLMs remains on solvable problems, but notably improves on unsolvable problems yet still falls short of solvable problems. However, small LLMs rarely show any progress despite employing reliable prompts. Therefore, we further propose an alignment strategy to enhance small LLMs' reliability, which can significantly improve LLM reliability performances on both in-domain and out-of-domain tasks.

Figures (26)

• Figure 1: Illustrations of (a) an unreliable LLM may fabricate incorrect or nonsensical content on math problems; (b) a reliable LLM can correctly answer solvable problems or identify unsolvable problems, or refuse to answer to avoid misleading users; (c) preference of LLM-generated responses.
• Figure 2: Illustrations of (a) a solvable question from AIME aimo2024aime and two rewritten schemes by removing one condition (b) or adding one contradictory condition (c).
• Figure 3: Unsolvable data construction workflow, where the example version is presented in Supplement 12.
• Figure 4: Results of Success Rate (Succ.) on o3-mini on different test sets (AIME, AMC, MATH, Minerva) on both solvable (slv) and unsolvable (usl) subsets using standard (std) and reliable (real) prompts, respectively.
• Figure 5: Illustrations of Succ.($\mathcal{U}$) of unsolvable problems regarding (a) two rewriting schemes (removal & contradiction), and (b) two difficulty levels labeled by experts (0: simple & 1: hard) on several typically used LLMs using reliable prompts. Completed results on all LLMs are presented in Supplement C.3.

Theorems & Definitions (1)

• Definition 1: LLM Reasoning Reliability