Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization

TL;DR

This work tackles the challenge of noisy feedback in iterative preference optimization for large language models by introducing Uncertainty-enhanced Preference Optimization (UPO). A dedicated estimator with Monte Carlo dropout in a Bayesian neural network quantifies pairwise uncertainty to sample reliable feedback, while an uncertainty-weighted self-evolution mechanism robustly updates policies via DPO. Across universal NLP benchmarks and mathematics reasoning tasks, UPO demonstrates strong improvements over SFT and standard DPO, achieving state-of-the-art auto-evaluation results and notable gains in reasoning and coding tasks. The approach reduces reliance on extensive manual labeling and enhances the robustness and quality of the LLM’s self-evolution through uncertainty-aware data selection and weighting.

Abstract

Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs), but the performance is still underwhelming due to too much noisy preference data yielded in the loop. To combat this issue, we present an \textbf{U}ncertainty-enhanced \textbf{P}reference \textbf{O}ptimization (UPO) framework to make the LLM self-evolve with reliable feedback. The key idea is mitigating the noisy preference data derived from the current policy and reward models by performing pair-wise uncertainty estimation and judiciously reliable feedback sampling. To reach this goal, we thus introduce an estimator model, which incorporates Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the preference data derived from the LLM policy. Compared to the existing methods that directly filter generated responses based on the reward score, the estimator focuses on the model uncertainty in a pair-wise manner and effectively bypasses the confirmation bias problem of the reward model. Additionally, we also propose an uncertainty-enhanced self-evolution algorithm to improve the robustness of preference optimization and encourage the LLM to generate responses with both high reward and certainty. Extensive experiments over multiple benchmarks demonstrate that our framework substantially alleviates the noisy problem and improves the performance of iterative preference optimization.

Figures (9)

• Figure 1: Overview of three paradigms.
• Figure 2: Illustration of UPO framework. We first use the labeled preference data to train a LLM policy, a reward model, and an estimator model. Then, multiple new preference data can be generated by the LLM policy based on the reward score. Finally, we use the uncertainty estimation technique to sample reliable data and further update the LLM policy with an uncertainty-enhanced self-evolution algorithm. The whole procedure repeats until convergence.
• Figure 3: The curve of training loss and LC win rate (%) on AlpacaEval 2.0 at each iteration.
• Figure 4: Performance of different iterations of UPO compared with SFT and DPO over MT-Bench.
• Figure 5: Noise rate (%) of different sampling strategies over multiple manual evaluation sets.