Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Lists to Emojis: How Format Bias Affects Model Alignment

Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, Tong Zhang

TL;DR

The work reveals pervasive format biases in RLHF, showing that preferences from humans, GPT-4, and leaderboards favor specific formats such as bold, lists, emojis, and links. It demonstrates that even tiny amounts of biased data can inject substantial bias into reward models, and that online alignment methods (DPO/PPO) amplify these patterns more than offline training. A two-head debiasing approach with a correlation loss is proposed and shown to reduce bias on RewardBench, though with trade-offs in reward quality and data sparsity issues. The findings argue for disentangling format and content in evaluation and alignment design to prevent reward hacking and improve generalization to real users.

Abstract

In this paper, we study format biases in reinforcement learning from human feedback (RLHF). We observe that many widely-used preference models, including human evaluators, GPT-4, and top-ranking models on the RewardBench benchmark, exhibit strong biases towards specific format patterns, such as lists, links, bold text, and emojis. Furthermore, large language models (LLMs) can exploit these biases to achieve higher rankings on popular benchmarks like AlpacaEval and LMSYS Chatbot Arena. One notable example of this is verbosity bias, where current preference models favor longer responses that appear more comprehensive, even when their quality is equal to or lower than shorter, competing responses. However, format biases beyond verbosity remain largely underexplored in the literature. In this work, we extend the study of biases in preference learning beyond the commonly recognized length bias, offering a comprehensive analysis of a wider range of format biases. Additionally, we show that with a small amount of biased data (less than 1%), we can inject significant bias into the reward model. Moreover, these format biases can also be easily exploited by downstream alignment algorithms, such as best-of-n sampling and online iterative DPO, as it is usually easier to manipulate the format than to improve the quality of responses. Our findings emphasize the need to disentangle format and content both for designing alignment algorithms and evaluating models.

From Lists to Emojis: How Format Bias Affects Model Alignment

TL;DR

The work reveals pervasive format biases in RLHF, showing that preferences from humans, GPT-4, and leaderboards favor specific formats such as bold, lists, emojis, and links. It demonstrates that even tiny amounts of biased data can inject substantial bias into reward models, and that online alignment methods (DPO/PPO) amplify these patterns more than offline training. A two-head debiasing approach with a correlation loss is proposed and shown to reduce bias on RewardBench, though with trade-offs in reward quality and data sparsity issues. The findings argue for disentangling format and content in evaluation and alignment design to prevent reward hacking and improve generalization to real users.

Abstract

In this paper, we study format biases in reinforcement learning from human feedback (RLHF). We observe that many widely-used preference models, including human evaluators, GPT-4, and top-ranking models on the RewardBench benchmark, exhibit strong biases towards specific format patterns, such as lists, links, bold text, and emojis. Furthermore, large language models (LLMs) can exploit these biases to achieve higher rankings on popular benchmarks like AlpacaEval and LMSYS Chatbot Arena. One notable example of this is verbosity bias, where current preference models favor longer responses that appear more comprehensive, even when their quality is equal to or lower than shorter, competing responses. However, format biases beyond verbosity remain largely underexplored in the literature. In this work, we extend the study of biases in preference learning beyond the commonly recognized length bias, offering a comprehensive analysis of a wider range of format biases. Additionally, we show that with a small amount of biased data (less than 1%), we can inject significant bias into the reward model. Moreover, these format biases can also be easily exploited by downstream alignment algorithms, such as best-of-n sampling and online iterative DPO, as it is usually easier to manipulate the format than to improve the quality of responses. Our findings emphasize the need to disentangle format and content both for designing alignment algorithms and evaluating models.
Paper Structure (43 sections, 8 equations, 4 figures, 8 tables)

This paper contains 43 sections, 8 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. The Pattern Bias in Preference Learning
  3. Pattern Statistics in Preference Datasets and Benchmarks
  4. Methods and datasets.
  5. Patterns.
  6. Results.
  7. Bias Evaluation of Preference Model
  8. Evaluation dataset for specific patterns.
  9. Evaluation results.
  10. Pattern Bias and Preference Flipping
  11. Pattern Bias Transfer in Downstream Reward Modeling and Alignment
  12. A Small Amount of Data Can Lead to Significant Reward Model Bias
  13. Base dataset.
  14. Attacking dataset.
  15. Model, and parameter.
  16. ...and 28 more sections

Figures (4)

  • Figure 1: We use the Llama-3-8B-it model as the base model and use the prompts from UltraFeedback to conduct DPO training. For bias evaluation, we generate one response for each question in AlpacaEval dataset and calculate the ratio of responses with bold/list pattern. While the base model already favors the bold and list patterns, the DPO training with the biased reward further amplifies these biases.
  • Figure 2: The test results on the AlpacaEval and split test set of UltraFeedback. We use the Llama-3-8B-SFT model as the base model and use the prompts from UltraFeedback to conduct iterative DPO and PPO training, where the reward model is used to annotate the on-policy samples from the model. Three reward models, trained in Section \ref{['sec:rm attack']} with varying sizes of attacking data, are used in this process. The offline DPO is a special case where we use all the prompts in one iteration. The online iterative DPO and PPO significantly amplify the pattern bias throughout training.
  • Figure 3: We use Llama-3-8B-it as the base model to generate n responses per prompt and use the different reward models to select the one with the highest reward. The ratio of samples containing bold pattern or list pattern increases as n grows in best-of-n sampling.
  • Figure :