Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control

Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum

TL;DR

This work proposes Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints, and introduces quantile-weighted FSD constraints, which provide a principled mechanism for tuning a model's risk profile via the quantile weighting function.

Abstract

Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events. This limitation is problematic when robustness and risk sensitivity are critical. Stochastic dominance offers a principled alternative by comparing entire cost distributions rather than just their averages, enabling direct control over tail risks and potential out-of-distribution failures that expectation-based constraints may overlook. In this work, we propose Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints. We operationalize this constraint by comparing the target policy's cost distribution to that of a reference policy within an Optimal Transport (OT) framework, using entropic regularization and Sinkhorn iterations to obtain a differentiable and computationally efficient objective for stable end-to-end optimization. Furthermore, we introduce quantile-weighted FSD constraints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs), so that improvements under weighted dominance imply guaranteed improvements in the corresponding spectral risk. This provides a principled mechanism for tuning a model's risk profile via the quantile weighting function. Empirical results demonstrate that RAD improves harmlessness over baselines while remaining competitive in helpfulness, and exhibits greater robustness on out-of-distribution harmlessness evaluations.

Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control

TL;DR

This work proposes Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints, and introduces quantile-weighted FSD constraints, which provide a principled mechanism for tuning a model's risk profile via the quantile weighting function.

Abstract

Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events. This limitation is problematic when robustness and risk sensitivity are critical. Stochastic dominance offers a principled alternative by comparing entire cost distributions rather than just their averages, enabling direct control over tail risks and potential out-of-distribution failures that expectation-based constraints may overlook. In this work, we propose Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints. We operationalize this constraint by comparing the target policy's cost distribution to that of a reference policy within an Optimal Transport (OT) framework, using entropic regularization and Sinkhorn iterations to obtain a differentiable and computationally efficient objective for stable end-to-end optimization. Furthermore, we introduce quantile-weighted FSD constraints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs), so that improvements under weighted dominance imply guaranteed improvements in the corresponding spectral risk. This provides a principled mechanism for tuning a model's risk profile via the quantile weighting function. Empirical results demonstrate that RAD improves harmlessness over baselines while remaining competitive in helpfulness, and exhibits greater robustness on out-of-distribution harmlessness evaluations.
Paper Structure (32 sections, 5 theorems, 45 equations, 2 figures, 5 tables)

This paper contains 32 sections, 5 theorems, 45 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Problem Setting
  3. Reinforcement Learning from Human Feedback (RLHF)
  4. Safe RLHF
  5. First-Order Stochastic Dominance
  6. Optimal Transport
  7. Relation to FSD.
  8. Spectral Risk Measures
  9. Risk-sensitive Alignment via Dominance
  10. Non-parametric (quantile-particle) representation of policy-induced cost distributions.
  11. Computing the RAD policy gradients.
  12. Universality of Weighted FSD for Controlling Spectral Risk Measures
  13. Empirical Results
  14. Model Evaluations
  15. Harmlessness
  16. ...and 17 more sections

Key Result

Theorem 1

Fix $\lambda\ge 0$ and define the dual objective $L(\theta,\lambda)$ by eq:sard_dual. Let $\alpha_1,\dots,\alpha_N\in(0,1)$ be fixed quantile levels and let $q_i(\pi_\theta):=Q_{C_{\pi_\theta}}(\alpha_i)$ denote the corresponding cost-quantile values. Using the empirical quantile-particle approximat Moreover, for the weighted dominance objective $\mathcal{L}^{w}_{\mathrm{FSD}}$ (see eq:weighted-fs

Figures (2)

  • Figure 1: Average reward win rates between competing models, across 3 seeds.
  • Figure 2: Illustration of spectral weighting functions corresponding to different spectral risk measures (SRMs). For parameterized families, the plots show how the spectral weights vary with the risk-aversion parameter $\lambda$. Note that VaR is represented as a gaussian with very small bandwidth instead of a dirac delta for implementation reasons. (YC: add the params used for wang in the plot)

Theorems & Definitions (5)

  • Theorem 1: RAD policy-gradient estimator
  • Proposition 2
  • Corollary 3
  • Theorem 4: from melnyk2024distributional
  • Corollary 5: FSD as asymmetric optimal transport