Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

Yanqi Dai, Yuxiang Ji, Xiao Zhang, Yong Wang, Xiangxiang Chu, Zhiwu Lu

TL;DR

MathForge presents a two-pronged framework to boost mathematical reasoning by focusing on harder questions. DGPO balances and prioritizes updates through MAD-based normalization (DGAE) and difficulty-aware weighting (DQW), while MQR systematically reformulates questions across Background, Term, and Sub-Problem to raise difficulty without altering the correct answer. Together, DGPO and MQR yield synergistic gains across multiple language models and benchmarks, with MathForge achieving state-of-the-art performance and demonstrating model- and modality-agnostic benefits. The work advances RLVR training by combining data-centric hardening with algorithmic emphasis on challenging problems, offering practical and scalable improvements for mathematical reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

TL;DR

MathForge presents a two-pronged framework to boost mathematical reasoning by focusing on harder questions. DGPO balances and prioritizes updates through MAD-based normalization (DGAE) and difficulty-aware weighting (DQW), while MQR systematically reformulates questions across Background, Term, and Sub-Problem to raise difficulty without altering the correct answer. Together, DGPO and MQR yield synergistic gains across multiple language models and benchmarks, with MathForge achieving state-of-the-art performance and demonstrating model- and modality-agnostic benefits. The work advances RLVR training by combining data-centric hardening with algorithmic emphasis on challenging problems, offering practical and scalable improvements for mathematical reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.
Paper Structure (31 sections, 2 theorems, 29 equations, 2 figures, 9 tables)

This paper contains 31 sections, 2 theorems, 29 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Difficulty-Aware Group Policy Optimization
  5. Difficulty-Balanced Group Advantage Estimation
  6. Difficulty-Aware Question-Level Weighting
  7. Multi-Aspect Question Reformulation
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Analysis of DGPO
  12. Analysis of MQR
  13. Related Work
  14. Conclusion
  15. The Use of Large Language Models (LLMs)
  16. ...and 16 more sections

Key Result

Theorem 1

Given a single question $q$ and its corresponding responses $\left\{o_i\right\}^G_{i=1}$, each query-response pair receives a binary accuracy reward $r_i \in \{0,1\}$, and $p$ represents the accuracy rate, i.e., the proportion for a reward of $1$. Then, the total update magnitude without clipping fo This total update magnitude varies with respect to the accuracy rate $p$, reaching its maximum when

Figures (2)

  • Figure 1: Training dynamics of DGPO vs. GRPO evaluated on the MATH500 benchmark. Both models are trained on MATH using Qwen2.5-Math-7B.
  • Figure 2: Training dynamics of Original vs. MQR on training and evaluation data. Both models are trained on MATH and evaluated on MATH500 using Qwen2.5-Math-7B.

Theorems & Definitions (4)

  • Theorem 1: Update Magnitude for a Single Question using GRAE
  • Theorem 2: Update Magnitude for a Single Question using DGAE
  • proof
  • proof