UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts

TL;DR

UTMath introduces a cutting-edge, OEIS-based benchmark of 1,053 problems across 9 mathematical domains, each with ~68 test cases, to rigorously evaluate mathematical reasoning via unit tests. It pairs the UTMath benchmark with the Reasoning-to-Coding of Thoughts (RCoT) prompting scheme, which separates reasoning and coding to foster deeper reasoning and more efficient solutions than traditional Program-of-Thought (PoT). Empirical results show that current LLMs perform modestly on UTMath (best around 32.6% pass) and that RCoT consistently improves pass rates and runtime efficiency across several models, highlighting the critical role of reasoning quality. The paper also releases UTMath-Train (>70k sequences) to support further research and provides detailed analyses of hard test cases, dataset construction, and category-wise performance, laying groundwork for future advances in mathematical reasoning and code synthesis.

Abstract

The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problems, existing benchmarks such as GSM8K and MATH present limitations, including narrow problem definitions with specific numbers and reliance on predetermined rules that hinder accurate assessments of reasoning and generality. This paper introduces the UTMath Benchmark, a robust evaluation framework designed to assess LLMs through extensive unit tests, with a focus on both the accuracy and generality of model responses. It comprises 1,053 cutting-edge problems spanning nine mathematical domains, with an average of 68 test cases per problem. UTMath is highly challenging, with the best-performing model, o1-mini, solving only 32.57\% of the problems, followed by o1-preview at 27.16\%, and GPT-4o at 26.93\%. Furthermore, we present the Reasoning-to-Coding of Thoughts (RCoT) approach, which encourages LLMs to engage in explicit reasoning prior to code generation, thereby facilitating the production of more sophisticated solutions and enhancing overall performance and efficiency. Additionally, we also release the UTMath-Train training dataset (more than 70k samples), to support the community in further exploring mathematical reasoning. Our benchmark can be accessed via the following link: https://github.com/UTMathGroup/UTMath

Figures (11)

• Figure 1: Comparison of UTMath with other benchmarks. On the left, GPT-4o easily solved one question but failed with a different numeric input. On the right, our benchmark is shown, where each problem includes multiple test cases, and a solution is correct only if all test cases are passed. We also propose a new prompting method RCoT in which the LLM first reasons through the problem and then generates code.
• Figure 2: RCoT significantly improves the efficiency and effectiveness of the solution. It indicates that our RCoT proves to be more effective, suggesting that it encourages the model to reason critically and find more efficient solutions.
• Figure 3: UTMath generation pipeline. After downloading 23,238 Principle Sequences from OEIS and cleaning the data, 1,053 usable sequences were obtained. Descriptions were standardized by adding background information and improving readability (highlighted in green, also shown in Appendix \ref{['sec:appendix:B2']}). Hard cases were introduced to enhance discriminative capability, including terms from later positions to prevent simplistic algorithms from passing.
• Figure 4: Performance comparison of models across PoT and RCoT tasks at different pass@k levels.
• Figure 5: Performance comparison between self-reasoning and using GPT-4o reasoning for coding across different models. The results show that models perform better when relying on GPT-4o's reasoning output.