Exploring and Addressing Reward Confusion in Offline Preference Learning

TL;DR

This paper shows that offline RLHF is susceptible to reward confusion, and proposes a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.

Abstract

Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.

Figures (9)

• Figure 1: Illustration of a simplified MDP (left) and reward confusion (right). The reward $R$ is a function of the feature $z_1$, but not $z_2$. Spurious correlation between $z_1$ and $z_2$ can cause a network to wrongly model $R$ as a function of $z_2$.
• Figure 1: Ground truth returns (mean ± standard deviation) for different methods on Confusing Minigrid. $\dag$ indicates a p-value of $\leq$ 0.1 (vs. baseline).
• Figure 2: The IMPEC algorithm creates a sorted preference chain of $n$ buckets, each containing one or more rollouts with equal returns.
• Figure 3: The failure percentages and their corresponding p-values of being significantly lower than the failure rate of the baseline
• Figure 4: The six Confusing Minigrid tasks. The tasks require agents to move to goal positions, go to doors, or fetch objects.