Diverse Preference Optimization

TL;DR

<3-5 sentence high-level summary> DivPO addresses the diversity collapse observed in post-training alignment of language models by modifying the selection of training pairs: instead of always promoting the highest-reward outputs, DivPO selects the most diverse high-reward candidate and the least diverse low-reward candidate using a configurable diversity criterion and reward threshold. The method yields higher diversity across structured and unstructured creative tasks, while maintaining or improving quality, in both offline and online training settings. It generalizes to arbitrary diversity criteria and can be integrated with existing preference optimization frameworks, offering a practical route to richer, more varied model outputs for creative generation and instruction following. The empirical results demonstrate substantial gains in diversity (e.g., up to ~74.6% more diverse than DPO) with competitive or improved quality across persona generation, five-word story prompts, full-story generation, and instruction-following benchmarks.

Abstract

Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. In this work we introduce Diverse Preference Optimization (DivPO), an optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and a 74.6% increase in story diversity, while maintaining similar win rates as standard baselines. On general instruction following, DivPO results in a 46.2% increase in diversity, and a 2.4% winrate improvement compared to DPO.

Figures (13)

• Figure 1: Diverse Preference Optimization (DivPO). We consider a diversity criterion $\mathcal{D}$ for selecting chosen and rejected responses from a pool in preference optimization. Rather than taking the highest rewarded response as the chosen ($y_c$), we select the most diverse response that meets a certain quality reward threshold. Similarly, the least diverse response that is below a threshold is selected as the rejected response ($y_r$). These are contrasted against each other to optimize both quality and diversity simultaneously.
• Figure 2: Persona Generation Statistics. Llama 3.1-8B-Instruct and DPO tend to repeatedly generate a small subset of names, as shown by frequency (%) of the top most frequently generated. In contrast, DivPO provides a substantially more uniform distribution over the most frequent attributes, in addition to overall improved diversity metrics (see \ref{['tab:persona_performance_comparison']}).
• Figure 3: Keyword Story Generation Results. We show quality (ArmoRM scores) vs Diversity (Unique 1-Grams) for $N$=16 responses per prompt. To tune diversity for baselines, we vary the baseline Lama 3.1-8B-Instruct temperature ($t$), and the DPO $\beta$ value. For our method, DivPO, we vary the $\rho$ hyperparamter for choosing preference pairs. Unless otherwise noted, all methods use $t$=1.
• Figure 4: Instruction for the structured persona generation task.
• Figure 5: Instruction template for the keyword story generation task.