Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models

TL;DR

This work addresses the reliability problem in Self-Rewarding Language Models (SRLM) arising from inconsistencies between internal reward signals. It introduces Self-Consistent Internal Rewards (SCIR), combining consistency training across internal reward models with dynamic consistency preference optimization to selectively use consistent data for Direct Preference Optimization. Empirical results on Mistral-7B show that SCIR yields meaningful gains in alignment metrics (notably AlpacaEval 2.0 length-controlled win rate and MT-Bench) and enhances reward modeling ability, compared to SRLM baselines and external reward models. The approach demonstrates improved robustness of self-generated preference data and offers a strategy to achieve better human-preference alignment without additional human annotations or external reward sources.

Abstract

Aligning Large Language Models (LLMs) with human preferences is crucial for their deployment in real-world applications. Recent advancements in Self-Rewarding Language Models suggest that an LLM can use its internal reward models (such as LLM-as-a-Judge) \cite{yuanself} to generate preference data, improving alignment performance without costly human annotation. However, we find that different internal reward models within the same LLM often generate inconsistent preferences. This inconsistency raises concerns about the reliability of self-generated preference data, hinders overall alignment performance, and highlights the need for further research to ensure reliable and coherent alignment with human preferences. To address this limitation, we propose Self-Consistent Internal Rewards (SCIR), a novel framework designed to enhance consistency among internal reward models during training. In each training step, we collect preference predictions from multiple pre-defined internal reward models and enforce consistency and confidence through an inconsistency penalty mechanism, thereby improving the reliability of these internal reward models. We selectively use data with consistent predictions for preference optimization, ensuring the quality of the preference data. By employing self-consistent internal rewards, our method significantly improves the alignment performance and reward modeling capability of LLMs, outperforming baseline methods by a notable margin.

Figures (3)

• Figure 1: An overview of our framework. For each iteration, the LLM $M_t$ generates responses for the prompts in the prompt pool, constructing unlabeled preference pairs. Then these pairs are used to optimize $M_t$ via Self-Consistent Intern Rewards (SCIR) Training. In each training step, the model's implicit DPO reward model and generative reward model predict the preference probabilities for each unlabeled preference pair. We use the consistency loss to encourage the preference probabilities of all internal reward models to be consistent. Meanwhile, preference pairs with consistent predictions across all internal reward models are selected for DPO optimization. The SCIR training results in model $M_{t+1}$, which is used for the next iteration.
• Figure 2: Consistency rate of internal reward models. $M_t$ is the model after the $t$ iterations. New Data and Trained Data refer to the preference data from the $t$-th and the ($t$-1)-th iteration, respectively.
• Figure 3: Results of the internal reward models on the subset of RewardBench. IRM is the implicit reward model and GRM is the generative reward model. Consistency means the IRM and GRM predict consistent preference labels.