FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
Yuyang Ding, Chi Zhang, Juntao Li, Haibin Lin, Xin Liu, Min Zhang
TL;DR
This paper tackles flawed-positive rollouts in RL with verifiable rewards for LLM reasoning, showing that such flaws can accelerate early learning but ultimately hinder reliability. It introduces Flawed-Aware Policy Optimization (FAPO), a parameter-free reward adjustment that uses a Generative Reward Model (GenRM) to detect flawed positives and to apply a process-level penalty that shifts optimization from warm-up to refinement. Empirically, FAPO improves outcome correctness, process reliability, and training stability across math and general-domain tasks without increasing token budgets, aided by an asynchronous GenRM–RL architecture. The work also discusses challenges like reward hacking and long-tail computation, offering infrastructure guidance for deploying GenRM in large-scale RL systems. Overall, FAPO provides a principled, scalable approach to balance rapid early gains with robust, reliable reasoning in RL for LLMs.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models (LLMs). In this context, models explore reasoning trajectories and exploit rollouts with correct answers as positive signals for policy optimization. However, these rollouts might involve flawed patterns such as answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are rewarded identically to fully correct ones, causing policy models to internalize these unreliable reasoning patterns. In this work, we first conduct a systematic study of flawed-positive rollouts in RL and find that they enable rapid capability gains during the early optimization stage, while constraining reasoning capability later by reinforcing unreliable patterns. Building on these insights, we propose Flawed-Aware Policy Optimization (FAPO), which presents a parameter-free reward penalty for flawed-positive rollouts, enabling the policy to leverage them as useful shortcuts in the warm-up stage, securing stable early gains, while gradually shifting optimization toward reliable reasoning in the later refinement stage. To accurately and comprehensively detect flawed-positive rollouts, we introduce a generative reward model (GenRM) with a process-level reward that precisely localizes reasoning errors. Experiments show that FAPO is effective in broad domains, improving outcome correctness, process reliability, and training stability without increasing the token budget.