FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization

TL;DR

FlipGuard addresses update regression in preference alignment for LLMs by introducing a reward-based focal constraint that activates only on negative flips, enforcing conditional congruence with the pre-aligned policy while preserving knowledge. The method derives a practical objective that blends the original alignment loss with a KL-based focal constraint, which, via KL$\leftrightarrow$CE equivalence, reduces to a cross-entropy term on flipped cases. It is evaluated with PPO and DPO across UltraFeedback, HH-RLHF, Summarization, and CVALUES, showing reduced negative flips, maintained or improved win rates, and preservation of SFT capabilities on academic benchmarks. The results demonstrate FlipGuard’s ability to mitigate update regression with limited hyperparameter tuning and across multiple base models, indicating practical viability for safer, more reliable alignment pipelines.

Abstract

Recent breakthroughs in preference alignment have significantly improved Large Language Models' ability to generate texts that align with human preferences and values. However, current alignment metrics typically emphasize the post-hoc overall improvement, while overlooking a critical aspect: regression, which refers to the backsliding on previously correctly-handled data after updates. This potential pitfall may arise from excessive fine-tuning on already well-aligned data, which subsequently leads to over-alignment and degeneration. To address this challenge, we propose FlipGuard, a constrained optimization approach to detect and mitigate update regression with focal attention. Specifically, FlipGuard identifies performance degradation using a customized reward characterization and strategically enforces a constraint to encourage conditional congruence with the pre-aligned model during training. Comprehensive experiments demonstrate that FlipGuard effectively alleviates update regression while demonstrating excellent overall performance, with the added benefit of knowledge preservation while aligning preferences.

Figures (6)

• Figure 1: Update regression in preference alignment. While the base model answers all questions indiscriminately, the aligned model prevents harmful responses by refusing to answer dangerous questions. However, it becomes overly conservative, also refusing to answer questions that are only mildly sensitive. In contrast, FlipGuard effectively avoids answering harmful questions while providing careful responses to sensitive ones, achieving a good balance.
• Figure 2: FlipGuard overview. The pipeline involves first customizing a reward characterization to measure the model's performance, then determining the premise of negative flips, and finally applying a focal distillation to encourage conditional congruence with the pre-aligned model during training.
• Figure 3: MT-Bench results for PPO and DPO with the design of FlipGuard, respecially.
• Figure 4: Token-level $D_{\text{KL}}(\pi_\theta||\pi_{\theta_0})$ of PPO and DPO on UltraFeedback during training.
• Figure 6: The effect of different $\gamma$ values. Experiments show that within a certain range, FlipGuard is not sensitive to the selection of $\gamma$ values.