Aligning Large Language Models from Self-Reference AI Feedback with one General Principle

TL;DR

The paper tackles scalable alignment of large language models by using self-reference AI feedback guided by a single general principle, aiming to overcome the limits of human labeling and position bias. It introduces a three-part methodology: preference labeling via self-referenced critique, self-consistency to debias annotations, and semantic perplexity-based quantification of preference strength to shape reward signals. Empirical results show improved reward-model accuracy across annotator sizes and substantial win-rate advantages for policy models trained with this feedback on standard benchmarks. This approach enhances scalability and robustness of AI-aligned systems for general assistant tasks, while acknowledging limitations such as annotator reliability and potential reward distribution shifts during RL.

Abstract

In aligning large language models (LLMs), utilizing feedback from existing advanced AI rather than humans is an important method to scale supervisory signals. However, it is highly challenging for AI to understand human intentions and societal values, and provide accurate preference feedback based on these. Current AI feedback methods rely on powerful LLMs, carefully designed specific principles to describe human intentions, and are easily influenced by position bias. To address these issues, we propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback under simple and general principles such as ``best for humanity``. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference, and finally determine which answer better fits human preferences according to the criticism. Additionally, we use a self-consistency method to further reduce the impact of position bias, and employ semantic perplexity to calculate the preference strength differences between different answers. Experimental results show that our method enables 13B and 70B Llama2-Chat annotators to provide high-quality preference feedback, and the policy models trained based on these preference data achieve significant advantages in benchmark datasets through reinforcement learning.

Figures (3)

• Figure 1: Illustration of the self-reference model feedback framework. The process begins with a user's query, which is fed into the policy model, $\pi_{\psi}^{SFT}$, to generate multiple candidate responses, labeled as A and B. The annotator trained with reinforcement learning will first generate a response to the query prompt and then evaluate the candidate responses based on its own response and pre-defined principle. These evaluation outcomes are then used to derive preference rankings. Then we use the annotator model to compute the text perplexity of each candidate response and determine the marginal scores of them.
• Figure 2: Our method compares to other baseline methods in terms of win rates assessed by GPT-4 on the evaluation dataset. All methods have different reward models during the reinforcement learning phase.
• Figure 3: Comparing the effects of the self-reference mechanism on the relative probabilities of option tokens across annotator models of varying sizes.