Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization

TL;DR

This work shows that negative chain-of-thought (CoT) trajectories, despite often yielding incorrect final answers, can significantly improve out-of-domain generalization in large language models. It analyzes data, optimization, and inference to reveal that negatives provide diverse, partially valid reasoning signals that act as a natural regularizer and promote exploratory reasoning. To leverage unfiltered data efficiently, the authors introduce GLOW, a gain-based loss weighting scheme that upweights underlearned samples based on inter-epoch progress, yielding robust cross-domain gains without discarding any trajectories. Empirically, GLOW improves cross-domain generalization on Qwen2.5-7B by 5.51% over positive-only SFT and boosts MMLU performance when used as an RL initialization to 76.47%, demonstrating the practical benefit of learning from negatives for both SFT and RL pretraining. Overall, the work advances data-efficient reasoning in LLMs by showing that negative supervision can broaden the reasoning space and enhance transfer across diverse tasks and domains.

Abstract

Supervised fine-tuning (SFT) on chain-of-thought (CoT) trajectories demonstrations is a common approach for enabling reasoning in large language models. Standard practices typically only retain trajectories with correct final answers (positives) while ignoring the rest (negatives). We argue that this paradigm discards substantial supervision and exacerbates overfitting, limiting out-of-domain (OOD) generalization. Specifically, we surprisingly find that incorporating negative trajectories into SFT yields substantial OOD generalization gains over positive-only training, as these trajectories often retain valid intermediate reasoning despite incorrect final answers. To understand this effect in depth, we systematically analyze data, training dynamics, and inference behavior, identifying 22 recurring patterns in negative chains that serve a dual role: they moderate loss descent to mitigate overfitting during training and boost policy entropy by 35.67% during inference to facilitate exploration. Motivated by these observations, we further propose Gain-based LOss Weighting (GLOW), an adaptive, sample-aware scheme that exploits such distinctive training dynamics by rescaling per-sample loss based on inter-epoch progress. Empirically, GLOW efficiently leverages unfiltered trajectories, yielding a 5.51% OOD gain over positive-only SFT on Qwen2.5-7B and boosting MMLU from 72.82% to 76.47% as an RL initialization.

Figures (11)

• Figure 1: (a) Qwen2.5-14B: SFT on positives improves in-domain math but transfers weakly to other reasoning tasks, whereas SFT on negatives yields broader cross-domain gains. Bars show final accuracy, and “+” indicates absolute improvement over the base model. (b) Qwen2.5-32B: training loss on MMLU. Red denotes positive-only SFT and blue denotes negative-only SFT. $\Delta$ is the per-sample inter-epoch loss difference.
• Figure 2: Token frequency differences between $M_{\text{neg}}$ and $M_{\text{pos}}$ on digits and high-entropy tokens.
• Figure 3: Detailed categorization of negative samples in OpenMathReasoning and MMLU.
• Figure 4: Ablation study on the hyperparameters $\alpha$ and $\beta$. GLOW exhibits stable performance across different settings, demonstrating the robustness of the reweighting formulation.
• Figure 5: Training loss of Qwen2.5 models and Llama3.1-8B on OpenMathReasoning (left) and MMLU (right). Losses drop across epochs, with the positive setting converging faster than the negative.

Theorems & Definitions (11)

• Lemma A.1: Loss reduction and gradient alignment
• proof
• Lemma A.2: Positive weight increments induce PSD augmentation
• proof
• Lemma A.3: Improvement on a $k$-dimensional subspace
• proof
• Lemma A.4: Fisher Hessian transfer on $U$
• proof
• Lemma A.5: Improved conditioning reduces parameter sensitivity
• Proposition A.6: Conditioning and generalization improvement