MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

TL;DR

This work addresses the challenge of improving mathematical reasoning in large language models by curating a large, math-centric pretraining corpus and introducing a novel method to generate math-focused code paired with natural-language reasoning. The authors build MathCode-Pile (19.2B tokens) by combining diverse math data sources with a model-generated, proof-backed code-then-reasoning dataset, enabling continued pretraining that yields substantial performance gains across multiple benchmarks. They demonstrate strong results for the MathCoder2 family, including post-trainingfinetuning that boosts tool-integrated reasoning capabilities, all while maintaining open-source data-processing and training pipelines for reproducibility. The study also provides thorough ablations and comparisons to existing datasets, underscoring the value of coupling reasoning steps with executable code for mathematical tasks.

Abstract

Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at https://github.com/mathllm/MathCoder2 .

Figures (6)

• Figure 1: The data processing pipeline. (a) shows the pipeline of prior works. (b) demonstrates our method. We first use a fastText classifier to filter the Common Crawl corpus, resulting in the initial filtered math texts. Then, we annotate part of the filtered texts to train a new fastText classifier, and conduct a second filtering, resulting in the finer filtered math texts. Then we use an instruction-tuned LLM to extract reasoning steps from these math-related texts, and translate the reasoning steps into corresponding code snippets. We execute the code snippets and compare the output with the expected result. If the code executes successfully and the result is as expected, the code is retained.
• Figure 2: The prompt for extracting reasoning steps from texts in the pretraining corpus and generating the corresponding Python snippets. {TEXT} is replaced with the text from the dataset collected in Sec. \ref{['sec:basic_data']}.
• Figure 3: The prompt for annotation of OpenWebMath and the initially filtered CC-En documents. {TEXT} is replaced with the content of the document.
• Figure 4: The prompt asking Llama-3.1-70B-Instruct to simply rewrite the text and improve its quality. {TEXT} is replaced with the content of the document.
• Figure 5: Comparison of the accuracy on GSM8K between adding and not adding mathematical code across different training steps.