WildReward: Learning Reward Models from In-the-Wild Human Interactions

TL;DR

This work demonstrates that reward models for LLMs can be trained directly from in-the-wild human interactions, circumventing the need for large annotated preference pairs. By constructing WildFB from WildChat with a two-stage noise-refinement process and training WildReward via ordinal regression, the authors achieve competitive or superior performance to conventional reward models while achieving strong calibration and cross-sample consistency. The approach benefits from user diversity and enables robust online DPO improvements, suggesting scalable real-world signals can drive reward modeling. The study highlights a practical path to scaling reward models alongside evolving human–AI interactions, with public release of code and data to foster further research.

Abstract

Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.

Figures (8)

• Figure 1: Illustration of a human-LLM interaction with implicit feedback signals in the conversation. The user provides valid feedback and identifies an error.
• Figure 2: Overview of the proposed pipeline for extracting human feedback from in-the-wild conversations.
• Figure 3: Performance on RM-Bench Normal across varying data sizes and user counts.
• Figure 4: Accuracy and data coverage against score difference threshold, filtering for predictions where the chosen-rejected score margin exceeds the threshold. The results are reported on RM-Bench Normal.
• Figure 5: ROC curves and ROC-AUC scores of different reward models in the pointwise evaluation.