Reward Learning from Multiple Feedback Types

TL;DR

This work investigates reward learning from multiple feedback types beyond traditional binary preferences. It defines six explicit feedback types, develops a synthetic generation pipeline, and trains reward models across ten RL environments to assess learning dynamics and downstream performance. The study shows that diverse feedback types can yield strong reward modeling and that joint modeling via ensembles can match or exceed single-type performance in some cases, while also revealing environment-dependent strengths and weaknesses. The findings highlight the potential of multi-type feedback to improve RLHF, with avenues for dynamic reward models, scaling to complex domains, and integrating human and AI feedback for more robust alignment.

Abstract

Learning rewards from preference feedback has become an important tool in the alignment of agentic models. Preference-based feedback, often implemented as a binary comparison between multiple completions, is an established method to acquire large-scale human feedback. However, human feedback in other contexts is often much more diverse. Such diverse feedback can better support the goals of a human annotator, and the simultaneous use of multiple sources might be mutually informative for the learning process or carry type-dependent biases for the reward learning process. Despite these potential benefits, learning from different feedback types has yet to be explored extensively. In this paper, we bridge this gap by enabling experimentation and evaluating multi-type feedback in a broad set of environments. We present a process to generate high-quality simulated feedback of six different types. Then, we implement reward models and downstream RL training for all six feedback types. Based on the simulated feedback, we investigate the use of types of feedback across ten RL environments and compare them to pure preference-based baselines. We show empirically that diverse types of feedback can be utilized and lead to strong reward modeling performance. This work is the first strong indicator of the potential of multi-type feedback for RLHF.

Figures (53)

• Figure 1: Generation of Simulated Feedback of different types: Based on an existing expert model and rollout buffer, we generate six types of feedback, including ratings and comparisons, demos and corrections, as well as descriptions and descriptive preferences.
• Figure 2: RL from individual feedback type reward models. The area boundaries indicate minimum and maximum values out of the sampled runs. Results are averaged over five random seeds/ feedback datasets. For full training curves see \ref{['app_subsec:rl_training_curves']}.
• Figure 3: Showcasing the influence of noise on different feedback types in the HalfCheetah-v5 environment.
• Figure 4: Correlation of Learned Reward Functions for Different Feedback Types: (a) Correlation differs between types (b) We do not observe a strong relationship between GT correlation and rewards.
• Figure 5: Robustness of different reward functions for different noise levels: With increasing noise levels, we see a drop in performance for types, but to a different degree.

Theorems & Definitions (2)

• Lemma B.1
• proof