Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models

Hyeonwoo Kim, Gyoungjin Gim, Yungi Kim, Jihoo Kim, Byungju Kim, Wonseok Lee, Chanjun Park

TL;DR

This work addresses the gap in mathematical reasoning in large language models by introducing SAAS, a sequential learning framework that first trains on Chain-of-Thought (CoT) reasoning and then on Program-of-Thought (PoT) with a cognitive retention mechanism. The approach aims to amplify solving ability by solidifying logical skills before delegating computation to code-like PoT reasoning, and by mixing CoT samples into the PoT phase to prevent cognitive forgetting. Extensive experiments on GSM8K, MATH, and related benchmarks show SAAS achieving state-of-the-art results across multiple model scales, with ablations highlighting the importance of sequencing and retention. The findings suggest that carefully ordered CoT-to-PoT training, safeguarded by cognitive retention, can substantially enhance both mathematical reasoning and computational accuracy in LLMs, offering a practical route toward more reliable mathematical problem solving in real-world applications.

Abstract

This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.

SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models

TL;DR

This work addresses the gap in mathematical reasoning in large language models by introducing SAAS, a sequential learning framework that first trains on Chain-of-Thought (CoT) reasoning and then on Program-of-Thought (PoT) with a cognitive retention mechanism. The approach aims to amplify solving ability by solidifying logical skills before delegating computation to code-like PoT reasoning, and by mixing CoT samples into the PoT phase to prevent cognitive forgetting. Extensive experiments on GSM8K, MATH, and related benchmarks show SAAS achieving state-of-the-art results across multiple model scales, with ablations highlighting the importance of sequencing and retention. The findings suggest that carefully ordered CoT-to-PoT training, safeguarded by cognitive retention, can substantially enhance both mathematical reasoning and computational accuracy in LLMs, offering a practical route toward more reliable mathematical problem solving in real-world applications.

Abstract

This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.
Paper Structure (21 sections, 2 equations, 4 figures, 3 tables)

This paper contains 21 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. SAAS: Solving Ability Amplification Strategy
  3. Chain-of-Thought Learning
  4. Program-of-Thought Learning
  5. Experiments
  6. Experimental Settings
  7. Dataset Details
  8. Training Details
  9. Model Details
  10. Evaluation Details
  11. Results and Analysis
  12. RQ1: Comparison with Competitors
  13. RQ2: Effectiveness of Sequential Learning and Cognitive Retention Strategy
  14. RQ3: Further Analysis of the Capabilities of SAAS
  15. RQ4: Case Study
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Overview of SAAS (Solving Ability Amplification Strategy) with two core strategies: i) sequential learning strategy; ii) cognitive retention strategy.
  • Figure 2: Overall procedure of the synthetic data generation.
  • Figure 3: Accuracies on GSM8K with respect to the number of required reasoning steps.
  • Figure 4: Responses of different learning approaches for a given question-answer pair.