Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning

TL;DR

The paper tackles the problem that not all negative samples are equally informative for improving LLM reasoning. It introduces Plausible Negative Samples (PNS), generated by a reverse GRPO framework guided by a composite reward that enforces format conformance, inverts correctness, and promotes high-quality reasoning traces. A Center-Regularized Bradley-Terry reward model and a four-part PNS reward (format, accuracy, RM, CoT) enable the generation of near-correct yet incorrect trajectories, which are then used as rejected data in Direct Preference Optimization. Across three backbones and seven math benchmarks, PNS yields consistent gains and better out-of-domain generalization, outperforming other negative-sample strategies, with ablations showing the critical role of the accuracy constraint and format gating. The results demonstrate that carefully crafted plausible negatives—dominated by subtle knowledge errors—strengthen model discrimination and reasoning capabilities with practical plug-and-play applicability.

Abstract

Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models.

Figures (6)

• Figure 1: Comparison of a correct sample (CS), a plausible negative sample (Pns), and a rejection sampling sample (RS).Pns produces an incorrect answer through deceptively coherent reasoning, receiving a higher RM score ($+3.34$) than even the correct solution ($+3.21$), while RS is easily detected ($-3.13$).
• Figure 2: Overview of plausible negative sample synthesis.Step 1: We construct preference pairs from sampled responses and train a Reward Model to assess reasoning quality. Step 2: We optimize a Policy Model via RL with the Pns Reward to generate responses with high reasoning quality but incorrect answers.
• Figure 3: Reward score distribution of our trained RM. The RM achieves 98.89% pairwise accuracy over $n{=}14{,}181$ pairs, with well-separated distributions for chosen (blue) and rejected (yellow) responses.
• Figure 4: Judgment accuracy on MATH-500 paired responses. Our RM-4B substantially outperforms both Qwen3-4B and GPT-4.1.
• Figure 5: Reward score distributions of different sample types.Pns closely aligns with CS (WD = 1.12), while RS concentrates in the negative range (WD = 3.70).