Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment

TL;DR

The paper tackles misalignment in reward-model-based LLM fine-tuning by treating the process as a conflict between the base policy and the proxy reward. It defines PACS and Kendall-Tau metrics to identify proxy-policy conflicts and introduces SHF-CAS to selectively solicit human feedback on high-conflict QA pairs, refining the reward function and policy. Across safety and helpfulness alignment tasks, SHF-CAS improves alignment as measured by gold rewards and reduced proxy-policy disagreement, outperforming baselines like RSO and random sampling, with GPT-4o-based supervision showing nuanced differences. The approach offers a principled, resource-efficient path to targeted refinement in RLHF and safer, more reliable LLM behavior.

Abstract

Reward-model-based fine-tuning is a central paradigm in aligning Large Language Models with human preferences. However, such approaches critically rely on the assumption that proxy reward models accurately reflect intended supervision, a condition often violated due to annotation noise, bias, or limited coverage. This misalignment can lead to undesirable behaviors, where models optimize for flawed signals rather than true human values. In this paper, we investigate a novel framework to identify and mitigate such misalignment by treating the fine-tuning process as a form of knowledge integration. We focus on detecting instances of proxy-policy conflicts, cases where the base model strongly disagrees with the proxy. We argue that such conflicts often signify areas of shared ignorance, where neither the policy nor the reward model possesses sufficient knowledge, making them especially susceptible to misalignment. To this end, we propose two complementary metrics for identifying these conflicts: a localized Proxy-Policy Alignment Conflict Score (PACS) and a global Kendall-Tau Distance measure. Building on this insight, we design an algorithm named Selective Human-in-the-loop Feedback via Conflict-Aware Sampling (SHF-CAS) that targets high-conflict QA pairs for additional feedback, refining both the reward model and policy efficiently. Experiments on two alignment tasks demonstrate that our approach enhances general alignment performance, even when trained with a biased proxy reward. Our work provides a new lens for interpreting alignment failures and offers a principled pathway for targeted refinement in LLM training.

Figures (2)

• Figure 1: Overview of our proposed SHF-CAS framework. Starting with a strong base policy and a possibly biased proxy reward model, we sample responses and identify high-conflict examples, cases where the proxy reward strongly disagrees with the base policy. These conflicts may reveal either complementary knowledge or shared ignorance. The selected high-conflict QA pairs are then sent for human feedback, which is used to refine the reward model and improve the policy via RM-based fine-tuning (e.g., RLHF). This process can be iterated to progressively enhance alignment quality.
• Figure 2: Illustration of the two key outcomes resulting from the interaction between the base policy and the proxy reward. The figure highlights regions of agreement, where both models align, and conflict, where the base policy and proxy reward diverge.