Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-objective Reinforcement learning from AI Feedback

Marcus Williams

TL;DR

MORLAIF tackles misalignment in reinforcement learning from AI feedback by decomposing human preferences into multiple principle-specific PMs (e.g., factuality, toxicity, sycophancy) trained with AI feedback and combined via scalarization to form a reward for PPO. The approach yields improved alignment over single-objective RLAIF across model sizes, and robust performance is observed regardless of the scalarization function used, suggesting the gains arise from preference decomposition itself. The paper demonstrates that smaller, principle-specific PMs can steer larger models effectively and discusses the trade-offs and potential scale limitations, along with interpretability and tunability advantages. Overall, MORLAIF offers a scalable, interpretable framework for aligning LLMs by modularizing reward modeling and enabling flexible adjustment of behavior without retraining PMs.

Abstract

This paper presents Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a novel approach to improving the alignment and performance of language models trained using reinforcement learning from AI feedback (RLAIF). In contrast to standard approaches that train a single preference model to represent all human preferences, MORLAIF decomposes this task into multiple simpler principles, such as toxicity, factuality, and sycophancy. Separate preference models are trained for each principle using feedback from GPT-3.5-Turbo. These preference model scores are then combined using different scalarization functions to provide a reward signal for Proximal Policy Optimization (PPO) training of the target language model. Our experiments indicate that MORLAIF outperforms the standard RLAIF baselines and that MORLAIF can be used to align larger language models using smaller ones. Surprisingly, the choice of scalarization function does not appear to significantly impact the results.

Multi-objective Reinforcement learning from AI Feedback

TL;DR

MORLAIF tackles misalignment in reinforcement learning from AI feedback by decomposing human preferences into multiple principle-specific PMs (e.g., factuality, toxicity, sycophancy) trained with AI feedback and combined via scalarization to form a reward for PPO. The approach yields improved alignment over single-objective RLAIF across model sizes, and robust performance is observed regardless of the scalarization function used, suggesting the gains arise from preference decomposition itself. The paper demonstrates that smaller, principle-specific PMs can steer larger models effectively and discusses the trade-offs and potential scale limitations, along with interpretability and tunability advantages. Overall, MORLAIF offers a scalable, interpretable framework for aligning LLMs by modularizing reward modeling and enabling flexible adjustment of behavior without retraining PMs.

Abstract

This paper presents Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a novel approach to improving the alignment and performance of language models trained using reinforcement learning from AI feedback (RLAIF). In contrast to standard approaches that train a single preference model to represent all human preferences, MORLAIF decomposes this task into multiple simpler principles, such as toxicity, factuality, and sycophancy. Separate preference models are trained for each principle using feedback from GPT-3.5-Turbo. These preference model scores are then combined using different scalarization functions to provide a reward signal for Proximal Policy Optimization (PPO) training of the target language model. Our experiments indicate that MORLAIF outperforms the standard RLAIF baselines and that MORLAIF can be used to align larger language models using smaller ones. Surprisingly, the choice of scalarization function does not appear to significantly impact the results.
Paper Structure (24 sections, 2 equations, 6 figures)

This paper contains 24 sections, 2 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Multi-Objective Reinforcement Learning and scalarization functions
  3. Methodology
  4. Preference Modeling:
  5. Reinforcement Learning from AI Feedback:
  6. Model details
  7. Evaluation
  8. Baselines
  9. Principles
  10. Code
  11. Results
  12. Preference Model Accuracy:
  13. Win Rates:
  14. Correlation Between the Principles and Ablation of Principles
  15. Related work
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Diagram illustrating the training setup.
  • Figure 2: (a) shows the accuracy the different models achieve when trained as a preference model for one of the different principles. (b) shows the accuracy of our Multi-Objective PMs for the different scalarization functions and our single objective baselines.
  • Figure 3: (a) shows the human judged win-rate of MORLAIF Llama-2-7b and one trained with GPT-2-medium PMs over the single objective version. (b) shows the GPT-4-Turbo judged win-rate for the different models.
  • Figure 4: Comparison between the win-rates for the different MORL scalarization functions. Note that $e_{i,j} = 1 - e_{j,i}$.
  • Figure 5: (a) shows the correlations between the feedback for the different principles. (b) shows how the multi-objective PM accuracy depends on the number of principles for GPT-2-medium, Llama-2-7b, and the theoretical ceiling which represents 100% accuracy on each individual principle.
  • ...and 1 more figures