Evaluating Feature Dependent Noise in Preference-based Reinforcement Learning

TL;DR

This work addresses the challenge of learning from preferences in reinforcement learning when teacher feedback is not perfectly reliable. It formalizes feature-dependent noise (FDN) and introduces multiple FD noise variants that tie label flips to trajectory features, similarity, and model uncertainty, including language-model-based feedback. Through extensive experiments on DMControl and Meta-World, the authors show that FDNs can significantly impair learning and that current denoisers like RIME can struggle to robustly filter such structured noise, while some non-denoising approaches remain competitive. The findings highlight the need for FDN-aware robustness methods and suggest that LLM/VLM feedback often exhibits FD-like characteristics, motivating future research on structured noise handling in PbRL.

Abstract

Learning from Preferences in Reinforcement Learning (PbRL) has gained attention recently, as it serves as a natural fit for complicated tasks where the reward function is not easily available. However, preferences often come with uncertainty and noise if they are not from perfect teachers. Much prior literature aimed to detect noise, but with limited types of noise and most being uniformly distributed with no connection to observations. In this work, we formalize the notion of targeted feature-dependent noise and propose several variants like trajectory feature noise, trajectory similarity noise, uncertainty-aware noise, and Language Model noise. We evaluate feature-dependent noise, where noise is correlated with certain features in complex continuous control tasks from DMControl and Meta-world. Our experiments show that in some feature-dependent noise settings, the state-of-the-art noise-robust PbRL method's learning performance is significantly deteriorated, while PbRL method with no explicit denoising can surprisingly outperform noise-robust PbRL in majority settings. We also find language model's noise exhibits similar characteristics to feature-dependent noise, thereby simulating realistic humans and call for further study in learning with feature-dependent noise robustly.

Figures (14)

• Figure 1: Examples of feature-dependent noise. A teacher may be prone to errors because of similarities (E1) or hidden details in the observation that are hard to notice (E2). We explore more types of FDN in our experiments.
• Figure 2: A diverse set of domains used in our experiments from DMControl and Meta-world.
• Figure 3: Each row is a domain: Walker-walk, HalfCheetah-run, Quadruped-walk. Curves show mean $\pm$ standard error over seeds; x-axis is Step, y-axis is Episodic return.
• Figure 4: Learning performance on VLM-sourced preferences on CartPole and Metaworld Soccer.
• Figure 5: Comparison over different algorithms in 8 types of 20% noise, in Cheetah Run.