Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimization

TL;DR

The paper addresses the limitation of human-like biases in LLM reasoning by introducing Confidence-Guided Reasoning Path Preference Optimization (CGPO), which uses a model's own confidence signal to identify maximal-uncertainty points and generate non-human-like reasoning paths. It formalizes a confidence-guided data construction pipeline and a DPO-like training objective, optimized over triplets $(s_{init}, s^+, s^-)$ with a token-level confidence metric. Empirically, CGPO yields consistent gains on mathematical reasoning benchmarks (e.g., up to +4.3 percentage points on GSM8K for MetaMath-Llama-8B) and code-generation tasks (LiveCodeBench +2.1%, LeetCode +4.0%), while relying on data produced by a small policy model rather than strong models or human annotations. The results demonstrate CGPO's scalability and generalization, including out-of-distribution performance on Omni-Math, indicating practical impact for enhancing LLM reasoning across domains; limitations include computational constraints and scope to code/math domains, with future work extending to larger models and broader tasks. $L_{ ext{CGPO}}( heta) = - \\mathbb{E}_{(s_{init}, s^+, s^-) \sim \mathcal{D}} [ \log \sigma( \beta(\Delta) ) ]$ where $\Delta = \Delta_\theta - \Delta_{\text{ref}}$ and $\Delta_\theta = \log \pi_{\theta}(s^+|s_{init}) - \log \pi_{\theta}(s^-|s_{init})$.$

Abstract

Current approaches for strengthening LLM reasoning tend to introduce a training bias toward human-like reasoning trajectories. In step-wise preference optimization, in particular, dependence on human or higher-capacity model annotations for intermediate steps limits exploration of alternative, non-human-like reasoning paths and thus constrains achievable performance. Furthermore, through a small-scale pilot study, we observed that in approximately 75% of cases, the model's first erroneous step occurs after the lowest-confidence point. This suggests that guiding the model at its lowest-confidence point before an error provides more accurate supervision than locating the first explicit error. In this paper, we propose Confidence-Guided Reasoning Path Preference Optimization (CGPO), a method that leverages a confidence signal to identify points of maximal uncertainty in the model's reasoning process and applies self-generated, non-human-like reasoning-path guidance to mitigate trajectory drift. Our experiments span diverse models applied to both code and mathematical reasoning tasks. The results show that, with the same amount of training data, our method using data generated by a small model can achieve better performance in most cases compared with approaches using data generated by a strong model or human-annotated.

Figures (5)

• Figure 1: The difference between human-like reasoning and non-human-like reasoning. Human-like methods emulate human cognitive processes and typically use predefined, readable anchors as delimiters to split the reasoning path. Nevertheless, it imposes substantial constraints on the content and format of data during sampling. In contrast, in this paper, non-human-like delineate reasoning steps are based on model confidence and provide reward supervision for tokens where the model is most uncertain, incurring lower cost and exhibiting broader applicability across various tasks.
• Figure 2: CGPO overview. The core idea of our method fails in two parts: 1) explore the non-human-like reasoning path by determining the model reasoning step using the initial policy model $\pi_0$, and 2) construct chosen/rejected pairs to train $\pi_0$ with step-wise DPO training objective.
• Figure 3: The relationship between the confidence threshold and the trained model performance.
• Figure 4: The relationship between the confidence threshold and the average number of tokens per pair.
• Figure 5: Examples where the lowest confidence appears before the first error.