Fairness Aware Reward Optimization
Ching Lam Choi, Vighnesh Subramaniam, Phillip Isola, Antonio Torralba, Stefanie Jegelka
TL;DR
Demographic biases in human preference data can propagate fairness defects into LLM alignment via reward models. Faro is an in-processing framework that enforces DP, EO, or CF constraints during reward training by a differentiable, proxy-based Lagrangian, yielding reward models that are ordinal, calibrated, and fair. The authors prove fairness certificates for the reward and show that these rewards transfer fairness to KL-regularized policy fine-tuning, with a non-empty Pareto frontier spanning accuracy and fairness trade-offs. Empirically, Faro reduces demographic bias and harmful generations across multiple LLMs and datasets while maintaining or improving task performance, offering a principled path toward fair-by-design LLMs.
Abstract
Demographic skews in human preference data propagate systematic unfairness through reward models into aligned LLMs. We introduce Fairness Aware Reward Optimization (Faro), an in-processing framework that trains reward models under demographic parity, equalized odds, or counterfactual fairness constraints. We provide the first theoretical analysis of reward-level fairness in LLM alignment, establishing: (i) provable fairness certificates for Faro-trained rewards with controllable slack; a (ii) formal characterization of the accuracy-fairness trade-off induced by KL-regularized fine-tuning, proving fairness transfers from reward to policy; and the (iii) existence of a non-empty Pareto frontier. Unlike pre- and post-processing methods, Faro ensures reward models are simultaneously ordinal (ranking correctly), cardinal (calibrated), and fair. Across multiple LLMs and benchmarks, Faro significantly reduces bias and harmful generations while maintaining or improving model quality.