InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

TL;DR

This paper tackles two fundamental flaws in existing LLM preference optimization—invariance to modeling choices and underutilization of pairwise comparative information. It introduces Intrinsic Self-reflective Preference Optimization (InSPO), deriving a globally optimal policy $\pi^*$ that conditions on both the context $x$ and an alternative response $y'$, and proving invariance to the scalarization $\Psi$ and reference policy $\pi_{\mathrm{ref}}$. InSPO is a plug-and-play enhancement for the DPO family that induces dense reward shaping during training and distills comparative reasoning into the policy, incurring zero inference overhead at deployment. Empirical results across multiple benchmarks and models show consistent improvements in win rates and length-controlled metrics, with ablations highlighting the benefits of symmetric cross-conditioning and longer context windows. The work advances robust, human-aligned LLMs by enabling intrinsic self-reflection without architectural changes or online rollout requirements.

Abstract

Direct Preference Optimization (DPO) and its variants have become standard for aligning Large Language Models due to their simplicity and offline stability. However, we identify two fundamental limitations. First, the optimal policy depends on arbitrary modeling choices (scalarization function, reference policy), yielding behavior reflecting parameterization artifacts rather than true preferences. Second, treating response generation in isolation fails to leverage comparative information in pairwise data, leaving the model's capacity for intrinsic self-reflection untapped. To address it, we propose Intrinsic Self-reflective Preference Optimization (\q), deriving a globally optimal policy conditioning on both context and alternative responses. We prove this formulation superior to DPO/RLHF while guaranteeing invariance to scalarization and reference choices. \q~serves as a plug-and-play enhancement without architectural changes or inference overhead. Experiments demonstrate consistent improvements in win rates and length-controlled metrics, validating that unlocking self-reflection yields more robust, human-aligned LLMs.

Figures (5)

• Figure 1: From pairwise preference to our proposed InSPO. Standard DPO (left) learns a suboptimal policy $\bar{\pi}$ from response comparisons where both the preferred and dispreferred responses are evaluated based solely on the prompt. InSPO (right) unleashes intrinsic self-reflection through symmetric cross-conditioning for learning an optimal policy $\pi^\ast$: the policy generates the preferred response while seeing the dispreferred one as context, and vice versa, allowing the model to leverage alternative responses as in-context guidance for improvement. Green terms highlight the self-reflection mechanism in our new objectives.
• Figure 2: Training dynamics of InSPO methods (solid) versus baselines (dashed). Our InSPO exhibits stable optimization with smooth loss convergence, consistent accuracy improvement, and enhanced reward margins without requiring additional on-policy rollouts.
• Figure 3: Generation comparison on AlpacaEval 2 between standard DPO and InSPO using Mistral-7B-Instruct. Compared to the baseline DPO model, the sequence-level model produces better structured responses with hierarchical organization, making the information more clearly presented and readable.
• Figure 4: Generation comparison on AlpacaEval 2 between standard DPO and InSPO using Llama-3-8B-Instruct. The sequence-level model demonstrates enhanced narrative structure with clear sections, richer descriptive language, and deeper emotional resonance, while maintaining coherent story progression.
• Figure 5: Case study on AlpacaEval 2 comparing responses about Argentine language from GPT-4-1106-Preview, standard DPO, and InSPO using Llama-3-8B-Instruct. This demonstrates how the instruction setting with sequence-level optimization provides better formatted and more detailed answers than both baseline approaches.

Theorems & Definitions (6)

• Proposition 3.1
• Theorem 3.2
• Theorem 4.1
• proof
• proof
• proof