The Realignment Problem: When Right becomes Wrong in LLMs

TL;DR

The paper tackles the Alignment-Reality Gap by reframing post-deployment alignment as a programmatic policy-editing problem. It introduces TRACE, a three-stage pipeline that triages existing preference data against a new policy, assigns alignment-impact weights to conflicts, and updates the model with a hybrid objective that inverts, punishes, and regularizes as needed. Through synthetic (SynthValueBench) and real-world (PKU-SafeRLHF) evaluations, TRACE demonstrates robust re-alignment across multiple model families with minimal utility loss, closely approaching gold-standard full re-training without new annotations. The work provides a scalable, dynamic framework for sustaining LLM alignment amid evolving norms and regulations, with broad practical implications for responsible AI deployment.

Abstract

The alignment of Large Language Models (LLMs) with human values is central to their safe deployment, yet current practice produces static, brittle, and costly-to-maintain models that fail to keep pace with evolving norms and policies. This misalignment, which we term the Alignment-Reality Gap, poses a growing challenge for reliable long-term use. Existing remedies are inadequate: large-scale re-annotation is economically prohibitive, and standard unlearning methods act as blunt instruments that erode utility rather than enable precise policy updates. We introduce TRACE (Triage and Re-align by Alignment Conflict Evaluation), a framework for principled unlearning that reconceives re-alignment as a programmatic policy application problem. TRACE programmatically triages existing preference data against a new policy, identifies high-impact conflicts via a alignment impact score, and applies a hybrid optimization that cleanly inverts, discards, or preserves preferences while safeguarding model performance. Empirical results show that TRACE achieves robust re-alignment across diverse model families (Qwen2.5-7B, Gemma-2-9B, Llama-3.1-8B). On both synthetic benchmarks and the PKU-SafeRLHF dataset under complex policy shift, TRACE enforces new principles without degrading general capabilities. Our work establishes a scalable, dynamic, and cost-effective paradigm for maintaining LLM alignment, providing a foundation for sustainable and responsible AI deployment.

Figures (10)

• Figure 1: Comparison of old and new policy responses to a mental health-related prompt without access to a new human oracle signal. The new policy enables constructive, therapeutically-framed advice, whereas the old policy requires complete avoidance.
• Figure 2: Old policy $\pi_\text{old}$ for SynthValueBench which contains 4 value dimensions to enable easy study of value transformations in a constraint space.
• Figure 3: New policy $\pi_\text{new}$ for SynthValueBench which contains trivial and non-trivial value dimension shifts for holistic alignment evaluation.
• Figure 4: Value principles of PKU-SafeRLHF dataset ($\pi_\text{old}$). The dataset contains 19 value dimensions enabling large scale value edits and non-trivial transformations.
• Figure 5: Complex non trivial shifts and transformations created on $\pi_\text{old}$ for the value dimensions defined in PKU-SafeRLHF ($\pi_\text{new}$).