Post-training Large Language Models for Diverse High-Quality Responses

TL;DR

This work tackles the diversity collapse observed in post-trained LLMs by introducing Diversity Quality Optimization (DQO), a principled training objective based on Determinantal Point Processes (DPPs) that jointly optimizes for semantic diversity and response quality. By embedding multiple candidate responses per prompt and maximizing the determinant of their kernel Gram matrix, DQO promotes linear independence in the semantic embedding space and balances diversity with quality via a reward-augmented embedding framework. The authors provide theoretical connections to D-Optimal Design, practical algorithmic solutions (including using L+I regularization and leave-one-out gradient estimators) to stabilize training, and extensive experiments demonstrating improved diversity without sacrificing task performance across reasoning, summarization, story generation, and instruction-following tasks. The approach yields robust improvements under various hyperparameters and decoding settings, suggesting meaningful practical impact for producing diverse and high-quality LLM outputs in real-world deployments, while acknowledging limitations such as reliance on embeddings and reward signals.”trailing_period

Abstract

Reinforcement learning (RL) has emerged as a popular method for post-training large language models (LLMs). While improving the model's performance on downstream tasks, it often reduces the model's output diversity, leading to narrow, canonical responses. Existing methods to enhance diversity are limited, either by operating at inference time or by focusing on surface-level differences. We propose a novel training method named DQO (Diversity Quality Optimization) based on determinantal point processes (DPPs) to jointly optimize LLMs for quality and semantic diversity. Our approach samples and embeds a group of responses for each prompt, then uses the determinant of a kernel-based similarity matrix to measure diversity as the volume spanned by the embeddings of these responses. DQO is flexible and can be applied on top of existing RL algorithms. Experiments across instruction-following, summarization, story generation, and reasoning tasks demonstrate that our method substantially improves semantic diversity without sacrificing model quality.

Figures (6)

• Figure 1: The quality and diversity scores of responses generated by the model trained with DQO compared to the model trained solely with reward. The reported quality score is the reward of one generation. The diversity score is the average across multiple diversity metrics, each computed over 10 responses. All responses are generated with a temperature of 1.0.
• Figure 2: The DPP-based diversity metric of DQO promotes meaningful semantic diversity while ensuring robustness agains degenerate solutions.
• Figure 3: From left to right: GRPO, DQO-pairwise distance, DQO-determinant. Each circle represents a different city, with the size proportional to the number of times it was recommended. For each model, we sampled 100 times with a temperature of 1.0.
• Figure 4: The performance of the trained model on $pass@n$ metrics. For DQO, we set hyperparameters $\alpha=1.0$ and $k=4$.
• Figure 5: The performance of the trained model on six diversity metrics. For each metric, the higher value means the higher diversity. For DQO, we set hyperparameters $\alpha=1.0$ and $k=4$. And the diversity metrics are calculated across $10$ generated responses per prompt.

Theorems & Definitions (2)

• Definition 1: $L$-ensemble
• Lemma 1