Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Models

Xiaomin Li, Xupeng Chen, Jingxuan Fan, Eric Hanchen Jiang, Mingye Gao

TL;DR

ENCORE identifies a robust negative correlation between safety-rule rating entropy and accuracy in predicting human preferences, and uses this to entropy-penalize rule aggregation. It introduces a simple, training-free weighting scheme where weights are $w_k \propto e^{-{\\mathcal{H}(\\psi_k)}/\\tau}$, leading to an aggregated reward $\\phi = \sum_k w_k \\psi_k$ that emphasizes reliable, low-entropy rules. Theoretical justification shows high-entropy rules yield negligible gradients under the Bradley–Terry loss, supporting the penalization. Empirically, ENCORE improves RewardBench safety task accuracy over baselines including random, uniform, single-rule, and LLM-as-a-judge, even with an 8B backbone, and demonstrates strong interpretability. It offers a practical, dataset-agnostic approach to multi-attribute reward modeling that can complement other weighting or rule-selection strategies.

Abstract

The safety alignment of large language models (LLMs) often relies on reinforcement learning from human feedback (RLHF), which requires human annotations to construct preference datasets. Given the challenge of assigning overall quality scores to data, recent works increasingly adopt fine-grained ratings based on multiple safety rules. In this paper, we discover a robust phenomenon: Rules with higher rating entropy tend to have lower accuracy in distinguishing human-preferred responses. Exploiting this insight, we propose ENCORE, a simple entropy-guided method to compose multi-head rewards by penalizing rules with high rating entropy. Theoretically, we show that such rules yield negligible weights under the Bradley-Terry loss during weight optimization, naturally justifying their penalization. Empirically, ENCORE consistently outperforms strong baselines, including random and uniform weighting, single-head Bradley-Terry, and LLM-as-a-judge, etc. on RewardBench safety tasks. Our method is completely training-free, generally applicable across datasets, and retains interpretability, making it a practical and effective approach for multi-attribute reward modeling.

ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Models

TL;DR

ENCORE identifies a robust negative correlation between safety-rule rating entropy and accuracy in predicting human preferences, and uses this to entropy-penalize rule aggregation. It introduces a simple, training-free weighting scheme where weights are , leading to an aggregated reward that emphasizes reliable, low-entropy rules. Theoretical justification shows high-entropy rules yield negligible gradients under the Bradley–Terry loss, supporting the penalization. Empirically, ENCORE improves RewardBench safety task accuracy over baselines including random, uniform, single-rule, and LLM-as-a-judge, even with an 8B backbone, and demonstrates strong interpretability. It offers a practical, dataset-agnostic approach to multi-attribute reward modeling that can complement other weighting or rule-selection strategies.

Abstract

The safety alignment of large language models (LLMs) often relies on reinforcement learning from human feedback (RLHF), which requires human annotations to construct preference datasets. Given the challenge of assigning overall quality scores to data, recent works increasingly adopt fine-grained ratings based on multiple safety rules. In this paper, we discover a robust phenomenon: Rules with higher rating entropy tend to have lower accuracy in distinguishing human-preferred responses. Exploiting this insight, we propose ENCORE, a simple entropy-guided method to compose multi-head rewards by penalizing rules with high rating entropy. Theoretically, we show that such rules yield negligible weights under the Bradley-Terry loss during weight optimization, naturally justifying their penalization. Empirically, ENCORE consistently outperforms strong baselines, including random and uniform weighting, single-head Bradley-Terry, and LLM-as-a-judge, etc. on RewardBench safety tasks. Our method is completely training-free, generally applicable across datasets, and retains interpretability, making it a practical and effective approach for multi-attribute reward modeling.

Paper Structure

This paper contains 43 sections, 1 theorem, 24 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-attribute Reward Models.
  4. Definitions and Notations
  5. Bradley-Terry.
  6. Fine-grained Rewarding.
  7. Multi-head Reward model.
  8. Reward model Evaluation.
  9. Reinforcement Learning from Human Feedback (RLHF).
  10. Discrete Entropy.
  11. Method
  12. Preliminary Experiments
  13. Safety Rules.
  14. Data.
  15. Correlation between Entropy and Accuracy.
  16. ...and 28 more sections

Key Result

Theorem 1

Consider pairwise preference learning with a Bradley-Terry loss. Let $z^{(i)} \in \{+1,-1\}$ indicate which of two responses is correct in the $i$-th sample $(x, y_A, y_B)$. Given a weighting vector ${\bm{w}}=(w_1,\dots,w_R)$ of the multi-head rewards, define as the reward margin combining rule‐specific ratings $\phi_k$. The per‐sample Bradley-Terry loss is and suppose the total loss is given by

Figures (4)

  • Figure 1: Pipeline of our ENCORE framework. Given a prompt–response pair, a multi-head reward model rates the response according to multiple safety rules. Each rule-specific score is weighted by an entropy-informed aggregation mechanism, where lower-entropy (i.e., more reliable) rules are assigned higher weights. The final reward is the weighted sum of rule-specific scores.
  • Figure 2: Negative Entropy and accuracy of 10 rules on HH, PKU, and the combined HH-PKU datasets.
  • Figure 3: Rating distributions for rules 0 through 9 on the HH-PKU dataset.
  • Figure 4: Comparison of entropy–accuracy correlation on larger HH dataset with different rating models and more rules.

Theorems & Definitions (1)

  • Theorem 1: High-entropy rule yields negligible weight