LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency

TL;DR

LEASE tackles the high sample cost of offline PbRL by combining a learned transition model for data augmentation with an uncertainty-aware, ensemble-based pseudo-labeling mechanism to grow labeled-like data while maintaining reward-model accuracy. It provides a state-action–level generalization bound for the reward model and a safe policy-improvement guarantee for the offline PbRL setting, enabling integration with standard offline RL solvers. Empirically, LEASE delivers comparable or superior performance to baselines using far fewer labeled preferences across D4RL MuJoCo and Adroit tasks, with clear benefits from data augmentation and the selection mechanism. The work highlights a practical, theory-grounded path to reducing human feedback burden in PbRL and offers insights transferable to model-based offline RL and broader offline learning contexts.

Abstract

Offline preference-based reinforcement learning (PbRL) provides an effective way to overcome the challenges of designing reward and the high costs of online interaction. However, since labeling preference needs real-time human feedback, acquiring sufficient preference labels is challenging. To solve this, this paper proposes a offLine prEference-bAsed RL with high Sample Efficiency (LEASE) algorithm, where a learned transition model is leveraged to generate unlabeled preference data. Considering the pretrained reward model may generate incorrect labels for unlabeled data, we design an uncertainty-aware mechanism to ensure the performance of reward model, where only high confidence and low variance data are selected. Moreover, we provide the generalization bound of reward model to analyze the factors influencing reward accuracy, and demonstrate that the policy learned by LEASE has theoretical improvement guarantee. The developed theory is based on state-action pair, which can be easily combined with other offline algorithms. The experimental results show that LEASE can achieve comparable performance to baseline under fewer preference data without online interaction.

Figures (6)

• Figure 1: The learning framework of LEASE. The offline dataset without reward is used to train the transition model, and a limited labeled preference dataset is used to pretrain the reward model. Then, the generated unlabeled preference data is screened through the uncertainty-aware selection mechanism. The reward model is updated based on the labeled and generated dataset. Finally, the agent learns the policy based on an offline dataset and a learned reward model.
• Figure 2: The description of Mujoco tasks (locomotion tasks) and Adroit tasks (manipulation tasks).
• Figure 3: The comparison between prediction value by the learned rewards and their ground truths for different methods under (a) hopper-medium-expert and (b) halfcheetah-medium-expert datasets, where the value predicted by the trained reward model and ground-truth reward value are both normalized to $[0,1]$. From left to right are methods LEASE, FEWER and FRESH, respectively.
• Figure 4: The relationship between pseudo-labeling accuracy and the number of preference dataset $N_l$.
• Figure : LEASE: offLine prEference-bAsed RL with high Sample Efficiency