Preference Elicitation for Offline Reinforcement Learning
Alizée Pace, Bernhard Schölkopf, Gunnar Rätsch, Giorgia Ramponi
TL;DR
This work tackles offline reinforcement learning when reward signals are unavailable by introducing offline preference elicitation via simulated rollouts (Sim-OPRL). It integrates model-based dynamics learning with pessimistic robust objectives for out-of-distribution states and an optimistic, information-seeking strategy for preference queries, yielding theoretical guarantees on sample complexity tied to offline data coverage of the optimal policy. The authors formalize the problem, provide bounds for both sampling-from-offline data (OPRL) and simulated-rollout strategies, and present a practical implementation with ensembles and uncertainty penalties. Empirically, Sim-OPRL surpasses offline preference baselines across diverse benchmarks, achieving better performance with fewer human queries. The approach has meaningful practical impact for safety-critical domains like healthcare, where environment interaction is restricted but expert input can be simulated or queried in a controlled setting.
Abstract
Applying reinforcement learning (RL) to real-world problems is often made challenging by the inability to interact with the environment and the difficulty of designing reward functions. Offline RL addresses the first challenge by considering access to an offline dataset of environment interactions labeled by the reward function. In contrast, Preference-based RL does not assume access to the reward function and learns it from preferences, but typically requires an online interaction with the environment. We bridge the gap between these frameworks by exploring efficient methods for acquiring preference feedback in a fully offline setup. We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm, which leverages a learned environment model to elicit preference feedback on simulated rollouts. Drawing on insights from both the offline RL and the preference-based RL literature, our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy. We provide theoretical guarantees regarding the sample complexity of our approach, dependent on how well the offline data covers the optimal policy. Finally, we demonstrate the empirical performance of Sim-OPRL in various environments.