Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models

Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, Dong Yu

TL;DR

SIaM introduces a code-based critic to guide self-improvement of LLMs for code-assisted mathematical reasoning. By iteratively generating, executing, and filtering code via the critic, and applying SFT and preference alignment on self-generated data from diverse web QA sources, the approach achieves robust gains in both in-domain and out-of-domain settings across English and Chinese. The critic also serves as a complementary evaluator, reducing reliance on hand-crafted heuristics. The results show meaningful improvements for 7–8B models, outperforming larger, prior code-assisted baselines on several benchmarks, and demonstrate strong cross-lingual generalization and robustness to diverse problem formats.

Abstract

There is a growing trend of teaching large language models (LLMs) to solve mathematical problems through coding. Existing studies primarily focus on prompting powerful, closed-source models to generate seed training data followed by in-domain data augmentation, equipping LLMs with considerable capabilities for code-aided mathematical reasoning. However, continually training these models on augmented data derived from a few datasets such as GSM8K may impair their generalization abilities and restrict their effectiveness to a narrow range of question types. Conversely, the potential of improving such LLMs by leveraging large-scale, expert-written, diverse math question-answer pairs remains unexplored. To utilize these resources and tackle unique challenges such as code response assessment, we propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. We also explore different alignment algorithms with self-generated instruction/preference data to foster continuous improvement. Experiments across both in-domain (up to +5.7%) and out-of-domain (+4.4%) benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.

SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models

TL;DR

SIaM introduces a code-based critic to guide self-improvement of LLMs for code-assisted mathematical reasoning. By iteratively generating, executing, and filtering code via the critic, and applying SFT and preference alignment on self-generated data from diverse web QA sources, the approach achieves robust gains in both in-domain and out-of-domain settings across English and Chinese. The critic also serves as a complementary evaluator, reducing reliance on hand-crafted heuristics. The results show meaningful improvements for 7–8B models, outperforming larger, prior code-assisted baselines on several benchmarks, and demonstrate strong cross-lingual generalization and robustness to diverse problem formats.

Abstract

There is a growing trend of teaching large language models (LLMs) to solve mathematical problems through coding. Existing studies primarily focus on prompting powerful, closed-source models to generate seed training data followed by in-domain data augmentation, equipping LLMs with considerable capabilities for code-aided mathematical reasoning. However, continually training these models on augmented data derived from a few datasets such as GSM8K may impair their generalization abilities and restrict their effectiveness to a narrow range of question types. Conversely, the potential of improving such LLMs by leveraging large-scale, expert-written, diverse math question-answer pairs remains unexplored. To utilize these resources and tackle unique challenges such as code response assessment, we propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. We also explore different alignment algorithms with self-generated instruction/preference data to foster continuous improvement. Experiments across both in-domain (up to +5.7%) and out-of-domain (+4.4%) benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
Paper Structure (22 sections, 7 equations, 1 figure, 20 tables)

This paper contains 22 sections, 7 equations, 1 figure, 20 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Training an Initial Model
  4. Building a Multi-Use Code-Based Critic Model
  5. Code Data Generation
  6. Self-Improvement with Unseen Data
  7. Experiments
  8. Data
  9. Implementation
  10. The Performance of the Initial Policy and Self-Improved LLMs
  11. The Comparison of Different Choices of Data and Alignment Methods
  12. Using the Critic Model as An Evaluator
  13. The Performance of Self-Improved LLMs on More Out-of-Domain Tasks
  14. Related Work
  15. Conclusions and Future Work
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Overview of our self-improving code-assisted paradigm using large-scale web QA data.