Hard Negative Sample-Augmented DPO Post-Training for Small Language Models

TL;DR

The paper targets the binary evaluation of mathematical reasoning in small language models by introducing a lightweight MathVerifier that decomposes reasoning into six interpretable dimensions and aggregates them into wrongness and absurdity scores. These signals are used for verifier-guided hard-negative mining and offline Direct Preference Optimization (DPO), avoiding large reward models or LLM judges. Starting from SFT on MetaMathQA, the approach yields targeted improvements on GSM8K and MATH for a 1.5B-parameter model, while maintaining realistic compute budgets. The work demonstrates that a carefully designed verifier and weighted offline preference learning can sharpen mathematical reasoning without online RLHF, suggesting a practical, transferable pipeline for structured reasoning tasks.

Abstract

Large language models (LLMs) continue to struggle with mathematical reasoning, and common post-training pipelines often reduce each generated solution to a binary outcome: correct or incorrect. This perspective is limiting in practice, as failures in chain-of-thought (CoT) reasoning are frequently structured; solutions may appear convincing while containing subtle logical, algebraic, or numerical flaws. Meanwhile, reinforcement learning from human feedback (RLHF) variants that rely on large reward models or LLM-as-a-judge signals are often expensive, difficult to scale, and unstable to iterate. We propose a lightweight and pragmatic post-training pipeline that targets such structured errors under realistic compute budgets. Starting from supervised fine-tuning (SFT) on MetaMathQA-style CoT data, we introduce a compact MathVerifier that decomposes a candidate solution into a six-dimensional error profile and aggregates it into interpretable wrongness and absurdity scores. These verifier signals serve two roles: (i) mining hard negatives that are near-correct yet structurally flawed, and (ii) defining per-sample importance weights that emphasize the most informative preference pairs. We integrate both into an offline Direct Preference Optimization (DPO) objective via a verifier-guided weighted formulation. Experiments on a 1.5B-parameter Qwen2.5 model show that verifier-guided, weighted DPO yields more targeted improvements than vanilla SFT and unweighted DPO, particularly on problems where solutions are numerically close to correct but logically inconsistent, while avoiding the overhead of training large reward models or relying on external judges.

Figures (5)

• Figure 1: Overall training pipeline. We first perform supervised fine-tuning (SFT) of Qwen2.5-1.5B on MetaMathQA with the standard next-token loss (Eq. \ref{['eq:sft-loss']}), obtaining $\pi_{\text{sft}}$. This model is then used to sample multiple chain-of-thought solutions for math benchmarks. The dual-channel MathVerifier assigns each trajectory a multi-dimensional error profile, aggregate wrongness/absurdity scores, and a raw importance weight. These signals drive active filtering and hard-negative mining, yielding weighted preference pairs $(y^+,y^-)$ used in a weighted DPO objective to train the final policy $\pi_\theta$, with a frozen reference policy $\pi_{\text{ref}}$.
• Figure 2: Training curves of the SFT base model used for all experiments (Qwen2.5-1.5B-Instruct fine-tuned on MetaMathQA).
• Figure 3: GSM8K exact match accuracy for the SFT baseline, random-uniform DPO, and verifier-guided hard-negative DPO across three model families. Bars show mean accuracy over seeds (five seeds for random DPO, three for hard-neg DPO). In every case, hard-negative DPO achieves higher accuracy than the corresponding random-DPO baseline on top of the same base model.
• Figure 4: MATH exact match accuracy for the same set of models as in Table \ref{['tab:gsm8k-main']}. As on GSM8K, verifier-guided hard-negative DPO consistently improves over the corresponding base model and typically outperforms random-uniform DPO, although random DPO can slightly degrade MATH performance in some cases.
• Figure 5: Training dynamics of DPO optimization and MathVerifier-guided reward shaping on the curated preference dataset.