Continuous-Utility Direct Preference Optimization

TL;DR

CU-DPO reframes reasoning alignment by replacing binary supervision with continuous utilities over a portfolio of $K$ strategies, enabling fine-grained assessment of reasoning quality. It introduces a two-phase training pipeline (Phase 1: strategy selection; Phase 2: execution refinement) and proves a $\Theta(K \log K)$ sample-efficiency advantage over binary DPO, along with convergence to an entropy-regularized utility-maximizing policy. Empirically, it achieves substantial gains in strategy selection accuracy and downstream reasoning on in-distribution math benchmarks, with scalable transfer to out-of-distribution tasks. The approach leverages strategy-conditioned chain sampling, an LLM-judged continuous utility, and margin-stratified pair construction to maximize informative supervision while avoiding conflicting signals. Overall, CU-DPO offers a generalizable framework for multi-strategy problem solving in complex domains, with potential extensions to coding, scientific reasoning, and planning tasks.

Abstract

Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce Continuous Utility Direct Preference Optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences, and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage training pipeline: (i) strategy selection, which optimizes the model to choose the best strategy for a given problem via best-vs-all comparisons, and (ii) execution refinement, which trains the model to correctly execute the selected strategy using margin-stratified pairs. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46 percent to 68-78 percent across seven base models, yielding consistent downstream reasoning gains of up to 6.6 points on in-distribution datasets with effective transfer to out-of-distribution tasks.

Figures (3)

• Figure 1: CU-DPO overview. Strategy-conditioned sampling $\rightarrow$ LLM-judged continuous utilities $\rightarrow$ progressive refinement $\rightarrow$ high-signal pair construction (Phase 1 [p1]: strategy selection, Phase 2 [p2]: execution refinement) $\rightarrow$ utility-weighted DPO training.
• Figure 2: Win-rate evolution per preference optimization step. Win rate versus fine-tuning steps for DeepMath, HARDMath2, and ProofNet. CU-DPO surpasses the baseline earlier and maintains a consistent advantage, demonstrating improved sample efficiency. Error bars indicate variability across evaluation batches and runs; the dashed line marks the 50% win-rate threshold (DeepSeek-R1-8B).
• Figure 3: Empirical evidence for reward--utility alignment. Learned implicit reward $r_\theta(x,y)=\beta(\log\pi_\theta-\log\pi_{\mathrm{ref}})$ aligns linearly with utility $U(x,y)$, supporting the relation $r_\theta(x,y)=U(x,y)+c(x)$ and Theorem \ref{['thm:dpo-convergence']}.

Theorems & Definitions (19)

• Theorem 3.1: Sample complexity under continuous utilities
• proof
• Theorem 3.2: Utility-guided sample efficiency gain
• proof
• Remark 3.3
• Lemma 3.4: Reward-utility alignment
• proof : Proof sketch
• Corollary 3.5: DPO converges to utility-maximizing policy
• proof
• Remark 3.6: Problem-dependent constant invariance