SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

TL;DR

This paper investigates how on-policy and off-policy data influence language model preference learning and demonstrates that they offer complementary strengths across task types. By isolating data source from the learning algorithm, the authors show that on-policy data better supports objective reasoning tasks while off-policy data enhances open-ended tasks. They propose SimpleMix, a straightforward method that blends on- and off-policy data during Direct Preference Optimization, achieving robust improvements over both single-source data and more complex hybrids. Across multiple models and benchmarks, SimpleMix yields meaningful gains, highlighting the practicality of simple data mixtures for LM alignment and guiding future data-curation strategies.

Abstract

Aligning language models with human preferences relies on pairwise preference datasets. While some studies suggest that on-policy data consistently outperforms off -policy data for preference learning, others indicate that the advantages of on-policy data may be task-dependent, highlighting the need for a systematic exploration of their interplay. In this work, we show that on-policy and off-policy data offer complementary strengths in preference optimization: on-policy data is particularly effective for reasoning tasks like math and coding, while off-policy data performs better on open-ended tasks such as creative writing and making personal recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach to combine the complementary strengths of on-policy and off-policy preference learning by simply mixing these two data sources. Our empirical results across diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it outperforms prior approaches that are much more complex in combining on- and off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.

Figures (11)

• Figure 1: Left panel: Our work studies data origin in preference optimization of LMs. Middle panel: We show that on-policy data and off-policy data are complementary: on-policy data mostly improves the model's performance on reasoning tasks that are objectively correct or incorrect (e.g., Math and Coding) while off-policy data improves on sub-tasks where humans might disagree with each other (e.g., creative writing and personal recommendation) (§ \ref{['Section:3']}). Right panel: Our proposed method SimpleMix mixes on and off-policy data, outperforming solely using either on or off-policy data. (§ \ref{['Section:4']})
• Figure 2: Comparison of win rates (against GPT-4-turbo) across different prompt categories in Alpaca Eval 2.0 for (left) objective tasks that have a groundtruth answer and (right) open-ended tasks where humans have individual preferences. On-policy DPO improves performance in math and coding, while off-policy DPO demonstrates better performance in creative writing and making personal recommendations.
• Figure 3: Comparison of win rates (against GPT-4-turbo) by the length of generation. On-policy training does not significantly outperform off-policy training as generation length increases in both math and coding tasks (left) and creative writing as well as recommendation tasks (right). Error bars show 95% confidence intervals from bootstrapping.
• Figure 4: Perfomance on Alpaca Eval 2.0 for different on- to off-policy data ratio for performing DPO on top of $\pi_\text{SFT} = \texttt{Meta-LLama-3.1-8B-Instruct} \text{ (left) and } \texttt{Llama-3.1-Tulu-8B-SFT}$ (right) on the Ultrafeedback pmlr-v235-cui24f dataset. A balanced mixture (0.5 on-policy + 0.5 off-policy) outperforms other mixtures.
• Figure 5: Alpaca Eval 2.0 length controlled win rate (LC) for different data selection strategies. No criterion other than selecting high-quality data (measured by a reward model) significantly outperforms others for off-policy preference learning.