Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning

Boyin Liu, Zhuo Zhang, Sen Huang, Lipeng Xie, Qingxu Fu, Haoran Chen, LI YU, Tianyi Hu, Zhaoyang Liu, Bolin Ding, Dongbin Zhao

TL;DR

This work addresses the instability caused by logical inconsistencies in AI feedback used for reinforcement learning from AI feedback (RLAIF). It introduces Conflict Detection Rate (CDR) to quantify preference conflicts and Deconflicted Graph Rewards (DGR) to transform conflicted judgments into a globally transitive reward signal by constructing a preference graph, extracting a directed acyclic graph via minimum feedback arc set, and computing deconflicted net-win scores. The method is modular and compatible with existing policy optimizers (e.g., GRPO, GSPO), and experiments demonstrate improved training stability and performance across multiple alignment benchmarks, with strong ablations confirming the value of principled cycle resolution. The findings emphasize that logical consistency in judge signals is a crucial, addressable factor in robust AI alignment, offering practical benefits and a foundation for further conflict-aware improvements in feedback-driven RL systems.

Abstract

Aligning language models using LLM judge feedback offers a scalable alternative to human annotation, yet is plagued by judgment inconsistencies that destabilize reinforcement learning. While prior work has focused on judge accuracy, the critical issue of logical coherence particularly preference cycles has been largely unaddressed. To address this gap, this work introduces an end to end framework to systematically detect and resolve these inconsistencies within the reinforcement learning training loop. Our framework features two core contributions: the Conflict Detection Rate (CDR), a novel metric to quantify judgment conflicts, and Deconflicted Graph Rewards (DGR), a signal-purification framework that eliminates cycles before policy optimization. DGR constructs preference graphs from raw judgments, transforms them into conflict-free Directed Acyclic Graphs (DAGs), and generates a logically coherent reward signal compatible with any policy optimizer. Experiments confirm that our framework significantly improves training stability and model performance over strong baselines, establishing logical consistency as a crucial and now-addressable dimension of AI feedback. The code for our method is available at https://github.com/modelscope/RM-Gallery.

Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning

TL;DR

This work addresses the instability caused by logical inconsistencies in AI feedback used for reinforcement learning from AI feedback (RLAIF). It introduces Conflict Detection Rate (CDR) to quantify preference conflicts and Deconflicted Graph Rewards (DGR) to transform conflicted judgments into a globally transitive reward signal by constructing a preference graph, extracting a directed acyclic graph via minimum feedback arc set, and computing deconflicted net-win scores. The method is modular and compatible with existing policy optimizers (e.g., GRPO, GSPO), and experiments demonstrate improved training stability and performance across multiple alignment benchmarks, with strong ablations confirming the value of principled cycle resolution. The findings emphasize that logical consistency in judge signals is a crucial, addressable factor in robust AI alignment, offering practical benefits and a foundation for further conflict-aware improvements in feedback-driven RL systems.

Abstract

Aligning language models using LLM judge feedback offers a scalable alternative to human annotation, yet is plagued by judgment inconsistencies that destabilize reinforcement learning. While prior work has focused on judge accuracy, the critical issue of logical coherence particularly preference cycles has been largely unaddressed. To address this gap, this work introduces an end to end framework to systematically detect and resolve these inconsistencies within the reinforcement learning training loop. Our framework features two core contributions: the Conflict Detection Rate (CDR), a novel metric to quantify judgment conflicts, and Deconflicted Graph Rewards (DGR), a signal-purification framework that eliminates cycles before policy optimization. DGR constructs preference graphs from raw judgments, transforms them into conflict-free Directed Acyclic Graphs (DAGs), and generates a logically coherent reward signal compatible with any policy optimizer. Experiments confirm that our framework significantly improves training stability and model performance over strong baselines, establishing logical consistency as a crucial and now-addressable dimension of AI feedback. The code for our method is available at https://github.com/modelscope/RM-Gallery.
Paper Structure (36 sections, 8 equations, 2 figures, 3 tables, 9 algorithms)

This paper contains 36 sections, 8 equations, 2 figures, 3 tables, 9 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. A Framework for Quantifying Preference Inconsistency
  5. DGR: Generating Consistent Rewards from Conflicted AI Feedback
  6. From Conflicted Preferences to Consistent Reward Signals
  7. Integrating DGR Signals with Policy Optimizers
  8. Experiments
  9. Experimental Setup
  10. Baseline Methods
  11. Comparative Evaluation on Alignment Benchmarks
  12. Ablation Studies
  13. Conflict Resolution Mechanism Analysis
  14. Robustness Across Different Reward Model Prompts
  15. Robustness and Scalability Analysis
  16. ...and 21 more sections

Figures (2)

  • Figure 1: Conflict detection and mitigation results. (a) CDR and accuracy comparison across different LLM judges. (b) Impact of prompt engineering on CDR reduction for Qwen3-32B. The pairwise prompts P1-P5 shown in the figure can be found in Appendix \ref{['sec:pairwise_prompts']}. Accuracy and conflict rates are computed on the non-tie subset of RewardBench2 malik2025rewardbench. See Appendix \ref{['sec:cdr_computation']} and \ref{['sec:accuracy_computation']} for detailed computation algorithms.
  • Figure 2: Sensitivity analysis of DGR compared to PREF on Arena-Hard. (a) Performance across three different LLM judges. (b) Performance as the number of candidates $n$ increases from 4 to 7.

Theorems & Definitions (1)

  • Definition 3.1: Preference Conflict