Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult

TL;DR

This work introduces Modulated Intervention Preference Optimization (MIPO), a per-instance alignment-aware refinement of preference optimization that modulates reference-model intervention based on how well a given data pair aligns with the reference. By measuring alignment with an average log-likelihood difference $K$ and using a modulator $q(K)=\log(1+e^K)$, MIPO reduces regularization for poorly aligned pairs to enable broader training while preserving alignment for well-aligned pairs. Empirical results on Mistral-7B and Llama3-8B across Alpaca Eval 2.0 and MT-Bench show that MIPO consistently outperforms DPO and approaches or surpasses SimPO in many settings, with robustness to the hyperparameter $\beta$ within a moderate range. The work highlights that average log-likelihood is not a universal alignment metric and discusses future directions for alternative adaptation terms and per-instance alignment assessment.

Abstract

Preference optimization methods typically begin training with a well-trained SFT model as a reference model. In RLHF and DPO, a regularization term is used during the preference optimization process to prevent the policy model from deviating too far from the reference model's distribution, thereby avoiding the generation of anomalous responses. When the reference model is already well-aligned with the given data or only requires slight adjustments, this approach can produce a well-aligned model. However, if the reference model is not aligned with the given data and requires significant deviation from its current state, a regularization term may actually hinder the model alignment. In this study, we propose \textbf{Modulated Intervention Preference Optimization (MIPO)} to address this issue. MIPO modulates the degree of intervention from the reference model based on how well the given data is aligned with it. If the data is well-aligned, the intervention is increased to prevent the policy model from diverging significantly from reference model. Conversely, if the alignment is poor, the interference is reduced to facilitate more extensive training. We compare the performance of MIPO and DPO using Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental results demonstrate that MIPO consistently outperforms DPO across various evaluation scenarios.

Figures (5)

• Figure 1: Alpacaeval 2.0 scores for MIPO and DPO implementations on Mistral-7B-Base and Llama-8B-Instruct. v0.1 is a model trained on different dataset.
• Figure 2: Optimization process of MIPO. In DPO, the objective utilizes a consistent regularization term (red part in DPO objective) for the reference model across all instances (A, B, C in Figure), regardless of the degree of alignment of each instance. However, in MIPO, the alignment of each instance with the reference model is first assessed by using the difference in average log likelihood. Based on this value, $K$, the extent to which the reference model will intervene in the learning process is determined and subsequently reflected in the MIPO objective.
• Figure 3: Alpaca Eval 2.0 scores in Mistral and Llama3 based on $\beta$. The dotted line represents the performance of DPO.
• Figure 4: The difference in average log likelihood changes after training for both MIPO and DPO, as applied to Mistral-7B-Base and Llama3-8B-Instruct.
• Figure :