Towards Federated RLHF with Aggregated Client Preference for LLMs

TL;DR

The paper addresses privacy concerns in collecting human preferences for RLHF by introducing federated RLHF methods FedBis and FedBiscuit. FedBis trains a binary selector in a federated setting to approximate preferred outputs, while FedBiscuit ensembles multiple selectors with cluster-wise aggregation to mitigate preference heterogeneity and reward hacking. A first federated RLHF benchmark on heterogeneous user preferences demonstrates that these methods improve professionalism and readability of generated content across summarization and QA tasks. The work also discusses integration with LoRA for efficiency and provides a framework for scalable, privacy-preserving alignment of LLMs with real user preferences, with practical implications for on-device or distributed deployments.

Abstract

Reinforcement learning with human feedback (RLHF) fine-tunes a pretrained large language model (LLM) using user preference data, enabling it to generate content aligned with human preferences. However, due to privacy concerns, users may be reluctant to share sensitive preference data. To address this, we propose utilizing Federated Learning (FL) techniques, allowing large-scale preference collection from diverse real-world users without requiring them to transmit data to a central server. Our federated RLHF methods (i.e., FedBis and FedBiscuit) encode each client's preferences into binary selectors and aggregate them to capture common preferences. In particular, FedBiscuit overcomes key challenges, such as preference heterogeneity and reward hacking, through innovative solutions like grouping clients with similar preferences to reduce heterogeneity and using multiple binary selectors to enhance LLM output quality. To evaluate the performance of the proposed methods, we establish the first federated RLHF benchmark with a heterogeneous human preference dataset. Experimental results show that by integrating the LLM with aggregated client preferences, FedBis and FedBiscuit significantly enhance the professionalism and readability of the generated content.

Figures (8)

• Figure 1: Comparison between standard and federated RLHF
• Figure 2: An outline of the proposed $\text{FedBis}$, an RLHF method in federated learning.
• Figure 3: Data distribution across different question domains on the selected clients.
• Figure 4: Prompt Templates.
• Figure 5: Gemma QA Sample.