TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations

TL;DR

Preference-based reinforcement learning (PbRL) suffers when human or VLM-provided preferences are noisy. TREND addresses this by a tri-teaching framework that trains three reward models in parallel and exchanges small-loss samples to denoise label data, augmented by few-shot expert demonstrations for improved initialization. On Meta-World robotic manipulation tasks, TREND remains robust up to 40% label noise and achieves high success rates, outperforming PEBBLE and the state-of-the-art RIME baselines, especially with noisy VLM feedback. The combination of cyclic peer learning for sample selection and minimal expert supervision provides a scalable, practical pathway to robust PbRL in real-world settings.

Abstract

Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning that relies on accurate preference labels. To address this challenge, we propose TREND, a novel framework that integrates few-shot expert demonstrations with a tri-teaching strategy for effective noise mitigation. Our method trains three reward models simultaneously, where each model views its small-loss preference pairs as useful knowledge and teaches such useful pairs to its peer network for updating the parameters. Remarkably, our approach requires as few as one to three expert demonstrations to achieve high performance. We evaluate TREND on various robotic manipulation tasks, achieving up to 90% success rates even with noise levels as high as 40%, highlighting its effective robustness in handling noisy preference feedback. Project page: https://shuaiyihuang.github.io/publications/TREND.

Figures (6)

• Figure 1: Overview of our method TREND. First, we pretrain the policy network using behavior cloning (BC) with few-shot expert demonstrations for effective exploration (A). In the online training phase, noisy preferences are collected from human annotators or a vision-language model (B1). We then apply our tri-teaching strategy for denoised reward learning, where three collaborative reward models identify clean preference samples for each other (B2). Finally, the learned reward model is used to guide the agent's training (B3), ensuring robust performance despite noisy labels.
• Figure 2: Learning curves for robot manipulation tasks on Meta-world. Each row represents results for a specific task and each column corresponds to a different error rate $\epsilon$. Results are averaged over five seeds. Shaded Areas represent standard deviation across seeds.
• Figure 3: Contribution of each component on Button-Press ($\epsilon=40\%$). Our tri-teaching boosts performance significantly, and adding a single expert demonstration further enhances success, emphasizing the need for both components.
• Figure 4: Results on Drawer-Open using VLM (Gemini-1.5-flash) to generate preference feedback. Our TREND-VLM-F achives the best result (left) under the high noise rate of VLM labels (right).
• Figure 5: (Left): Comparison between our method (w. Tri-teach) and a baseline denoising strategy (w. Self-teach) on Hammer ($\epsilon=40\%$). (Right): Comparison of clean label ratio under different noise levels between our method (w. Tri-teach) and the baseline (w/o. Tri-teach) on Hammer.