Deep Reinforcement Learning from Hierarchical Preference Design

TL;DR

This work tackles reward design in reinforcement learning by introducing HERON, a hierarchical preference-based framework that learns rewards from trajectory comparisons organized by an expert-defined hierarchy over multiple feedback signals. By eliciting preferences through a decision tree and training a neural reward model with a Bradley-Terry loss, HERON guides policy optimization without relying on hand-crafted linear reward aggregates. Across traffic control, code generation, language-model alignment, and robotics, HERON achieves superior or robust performance relative to reward-engineering baselines and demonstrates the benefits of scale-invariant, hierarchical signaling. The approach enhances sample efficiency and alignment capabilities, with potential extensions to direct preference optimization and richer hierarchies.

Abstract

Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such as improved sample efficiency and robustness. Our code is available at \url{https://github.com/abukharin3/HERON}.

Figures (14)

• Figure 1: (a) Evaluation curves with different reward hierarchies in traffic light control. The curve is within $\pm$ one standard deviation. (b) Utilization of different signals.
• Figure 2: Evaluation curves with different reward hierarchies in traffic light control. The importance decreases from left to the right in a label. The curve is within $\pm$ one standard deviation.
• Figure 3: Evaluation curves with different environments: changes of vehicles' speed limit. The baseline speed limit is 35 MPH. The curves are within $\pm$ one standard deviation.
• Figure 4: Training time and ablation study for HERON.
• Figure 5: Tuning cost of HERON in the traffic light control task.