Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback

TL;DR

This work tackles robust preference-based reinforcement learning when feedback comes from multiple, heterogeneous experts. It introduces TriTrust-PBRL (TTP), a joint framework that learns a shared reward function and per-expert trust parameters, which can be positive, near-zero, or negative, enabling automatic inversion of adversarial preferences. The authors provide identifiability analysis showing the model is recoverable up to an affine transformation under reasonable data connectivity assumptions, along with gradient dynamics explaining how trustworthy, noisy, and adversarial experts separate during training. Empirically, TTP delivers state-of-the-art robustness on manipulation and locomotion benchmarks (MetaWorld and DMControl), maintaining near-oracle performance under adversarial corruption and outperforming baselines when learning from mixed expert pools; results highlight the importance of explicitly modeling expert reliability for efficient and reliable PBRL in realistic multi-expert settings.

Abstract

Preference-based reinforcement learning (PBRL) offers a promising alternative to explicit reward engineering by learning from pairwise trajectory comparisons. However, real-world preference data often comes from heterogeneous annotators with varying reliability; some accurate, some noisy, and some systematically adversarial. Existing PBRL methods either treat all feedback equally or attempt to filter out unreliable sources, but both approaches fail when faced with adversarial annotators who systematically provide incorrect preferences. We introduce TriTrust-PBRL (TTP), a unified framework that jointly learns a shared reward model and expert-specific trust parameters from multi-expert preference feedback. The key insight is that trust parameters naturally evolve during gradient-based optimization to be positive (trust), near zero (ignore), or negative (flip), enabling the model to automatically invert adversarial preferences and recover useful signal rather than merely discarding corrupted feedback. We provide theoretical analysis establishing identifiability guarantees and detailed gradient analysis that explains how expert separation emerges naturally during training without explicit supervision. Empirically, we evaluate TTP on four diverse domains spanning manipulation tasks (MetaWorld) and locomotion (DM Control) under various corruption scenarios. TTP achieves state-of-the-art robustness, maintaining near-oracle performance under adversarial corruption while standard PBRL methods fail catastrophically. Notably, TTP outperforms existing baselines by successfully learning from mixed expert pools containing both reliable and adversarial annotators, all while requiring no expert features beyond identification indices and integrating seamlessly with existing PBRL pipelines.

Figures (16)

• Figure 1: Overview of the TriTrust-PBRL (TTP) framework. TTP jointly learns a shared reward network $R(\tau)$ and expert-specific trust parameters $\{\alpha_k\}_{k=1}^K$ from heterogeneous preference feedback. Reliable experts (with consistent correct preferences) develop positive trust parameters ($\alpha_k > 0$), noisy experts (with inconsistent random feedback) evolve toward zero ($\alpha_k \approx 0$), and adversarial experts (with systematically flipped preferences) develop negative trust parameters ($\alpha_k < 0$). During gradient updates, the weighted loss components are scaled by each expert's trust: reliable experts are up-weighted, noisy experts are down-weighted, and adversarial experts are inverted and weighted. The confidence-weighted joint loss (Eq. \ref{['eq:nll']}) aggregates all expert feedback while automatically adapting to their reliability, enabling robust reward learning even under adversarial corruption.
• Figure 2: Experimental environments used in this work. We evaluate on two locomotion domains from DMControl (top row) and two manipulation domains from MetaWorld (bottom row).
• Figure 3: Evaluation success rate for MetaWorld Sweep-Into-v2 under adversarial conditions ($\beta=[1,1,1,-1]$).
• Figure 4: Learned trust trajectories on Sweep-Into-v2 under adversarial conditions.
• Figure 5: Evaluation success rate for MetaWorld Sweep-Into-v2 under noisy conditions ($\beta=[1,1,1,0]$).

Theorems & Definitions (7)

• Lemma 1: Logistic identifiability
• proof
• Theorem 1: Identifiability
• proof
• Corollary 1
• proof
• Remark 1