Corruption Robust Offline Reinforcement Learning with Human Feedback
Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla, Goran Radanović
TL;DR
This work studies corruption-robust offline reinforcement learning from human feedback (RLHF) under a Huber contamination model where an $\varepsilon$-fraction of data may be corrupted. It develops a reduction-based framework that robustifies reward model learning via robust logistic regression, constructs a confidence set around the reward, and performs pessimistic offline RL over that set. The authors provide provable guarantees across three data-coverage regimes—Uniform Coverage, Low Relative Condition Number, and Bounded Generalized Coverage Ratio—achieving suboptimality that scales with $\varepsilon$ (up to log factors) and exploiting both zero-order and first-order offline RL oracles. By fusing PbRL with corruption-robust offline RL, the paper delivers principled, near-optimal strategies for RLHF under adversarial or noisy preferences. It also lays groundwork for future work on extending these results to general function approximation and trajectory-based rewards.
Abstract
We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting. Given an offline dataset of pairs of trajectories along with feedback about human preferences, an $\varepsilon$-fraction of the pairs is corrupted (e.g., feedback flipped or trajectory features manipulated), capturing an adversarial attack or noisy human preferences. We aim to design algorithms that identify a near-optimal policy from the corrupted data, with provable guarantees. Existing theoretical works have separately studied the settings of corruption robust RL (learning from scalar rewards directly under corruption) and offline RLHF (learning from human feedback without corruption); however, they are inapplicable to our problem of dealing with corrupted data in offline RLHF setting. To this end, we design novel corruption robust offline RLHF methods under various assumptions on the coverage of the data-generating distributions. At a high level, our methodology robustifies an offline RLHF framework by first learning a reward model along with confidence sets and then learning a pessimistic optimal policy over the confidence set. Our key insight is that learning optimal policy can be done by leveraging an offline corruption-robust RL oracle in different ways (e.g., zero-order oracle or first-order oracle), depending on the data coverage assumptions. To our knowledge, ours is the first work that provides provable corruption robust offline RLHF methods.