Forward versus Backward: Comparing Reasoning Objectives in Direct Preference Optimization

TL;DR

Large language models often produce plausible but incorrect reasoning; this paper examines how two training objectives—forward reasoning traces and backward verification traces—affect reasoning accuracy and error acknowledgement when optimized with Direct Preference Optimization. Using GSM8K and LoRA-tuned LLaMA 3.1 8B-Instruct, it shows forward-only training yields the largest accuracy gains, while backward-only training mainly improves verification calibration; both approaches reduce acknowledgement rate. Acknowledging rate is introduced as a key metric for error-awareness in reasoning. The authors provide a complete, reproducible training and evaluation pipeline and argue that forward and backward objectives yield complementary signals for building more reliable, calibrated reasoning systems.

Abstract

Large language models exhibit impressive reasoning capabilities yet frequently generate plausible but incorrect solutions, a phenomenon commonly termed hallucination. This paper investigates the effect of training objective composition on reasoning reliability through Direct Preference Optimization. Two complementary training signals are examined: forward chain-of-thought generation, which trains the model to produce correct reasoning traces, and backward verification, which trains the model to verify and acknowledge errors in candidate solutions. Experiments on GSM8K reveal a fundamental trade-off between these objectives. Forward-only DPO training achieves the highest accuracy improvement, increasing from 83.1% to 86.6% (+3.5 percentage points), while backward-only training yields minimal accuracy gains but substantially reduces the false positive rate from 13.4% to 4.3%. Notably, both training variants reduce acknowledgement rate compared to the baseline, suggesting that preference optimization increases model confidence in its outputs. These findings indicate that forward and backward reasoning objectives provide distinct and complementary learning signals: forward training improves problem-solving capability, while backward training improves verification calibration. The complete training and evaluation pipeline, implemented efficiently through Low-Rank Adaptation, is released to facilitate further research.