SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

TL;DR

This work tackles the data bottleneck in reinforcement learning with verifiable rewards (RLVR) for LLM reasoning by introducing SwS, a Self-aware Weakness-driven Problem Synthesis framework. SwS identifies model weaknesses during a preliminary RL phase, extracts core concepts from failure cases, and synthesizes targeted problems that align with the model's learning gaps, then augments training with these problems. Across 3B–32B models and eight benchmarking tasks, SwS yields substantial gains (e.g., +10.0% for 7B and +7.7% for 32B) and reduces the number of unresolved failures in key domains, demonstrating improved data efficiency and generalization. The paper also extends SwS with Weak-to-Strong Generalization, Self-evolving, and Weakness-driven Selection to show robustness and adaptability of weakness-focused data synthesis in diverse RLVR settings.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and limited-verification answers in existing distillation-oriented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model's capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a Self-aware Weakness-driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses as questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure cases and synthesize new problems to strengthen the model's weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization byempowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks.

Figures (12)

• Figure 1: 32B model performance across mainstream reasoning benchmarks and different domains.
• Figure 2: Illustration of the self-aware weakness identification during a preliminary RL training.
• Figure 3: An overview of our proposed weakness-driven problem synthesis framework that targets at mitigating the model's reasoning limitations within the RLVR paradigm.
• Figure 4: The ratios of consistently failed problems from different categories in the MATH-12k training set under different training configurations. (Base model: https://huggingface.co/Qwen/Qwen2.5-7B).
• Figure 5: Comparison of accuracy improvements using (a) Pass@1 on full benchmarks evaluated in Table \ref{['table:performance']} and (b) Avg@32 on the competition-level benchmarks. (c) illustrates the proportion of prompts within a batch that achieved 100% correctness across multiple rollouts during training.