Exploring Re-inforcement Learning via Human Feedback under User Heterogeneity
Sarvesh Shashidhar, Abhishek Mishra, Madhav Kotecha
TL;DR
The paper tackles RLHF under user heterogeneity by clustering annotators in embedding space and learning cluster-specific reward models. It proposes a joint optimization framework that simultaneously learns cluster assignments and cluster-specific reward parameters, evaluated on the Reddit TL;DR dataset. Results show that personalized reward models per cluster can achieve higher alignment (win-rate) than naive, homogeneous RLHF, demonstrating the value of accounting for heterogeneous preferences. This work lays groundwork for more scalable, domain-aware personalization of reward models in human-in-the-loop AI systems.
Abstract
Re-inforcement learning from human feedback (RLHF) has been effective in the task of AI alignment. However, one of the key assumptions of RLHF is that the annotators (referred to as workers from here on out) have a homogeneous response space. This assumption is not true in most practical settings and there have been studies done in the past to challenge this notion. This work has been inspired by such studies and explores one of the ways to deal with heterogeneity in worker preferences - by clustering workers with similar preferences and personalising reward models for each cluster. This work provides an algorithm that encourages simultaneous learning of reward models and worker embeddings. This algorithm is then empirically tested against the Reddit TL;DR dataset with unique worker IDs. We have shown that clustering users into different groups based on their preferences and created personalised reward models improves win-rate of the said models. Along with results and visualisations, this work aims to act as a stepping stone to more complicated models and gives a list of possible future extensions.