WorldPM: Scaling Human Preference Modeling

TL;DR

WorldPM demonstrates that human preference modeling exhibits scaling laws similar to language modeling, with objective and adversarial metrics improving with model size and data, while subjective metrics are less scalable due to style biases. Using StackExchange-derived preferences, the authors train 15M pairs on models up to 72B and show that WorldPM enhances downstream preference fine-tuning and RLHF-based alignment, especially with larger models and when data is limited. They also introduce style-content separation in evaluation to mitigate style bias and analyze training dynamics, revealing an early 'moment of epiphany' in optimization. The work highlights WorldPM as a scalable foundation for preference modeling, offering practical benefits for alignment pipelines and suggesting directions for refining subjective evaluation and data sources.

Abstract

Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling. We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential, where World Preference embodies a unified representation of human preferences. In this paper, we collect preference data from public forums covering diverse user communities, and conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters. We observe distinct patterns across different evaluation metrics: (1) Adversarial metrics (ability to identify deceptive features) consistently scale up with increased training data and base model size; (2) Objective metrics (objective knowledge with well-defined answers) show emergent behavior in larger language models, highlighting WorldPM's scalability potential; (3) Subjective metrics (subjective preferences from a limited number of humans or AI) do not demonstrate scaling trends. Further experiments validate the effectiveness of WorldPM as a foundation for preference fine-tuning. Through evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly improves the generalization performance across human preference datasets of varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5% on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we observe significant improvements on both in-house and public evaluation sets, with notable gains of 4% to 8% in our in-house evaluations.

Figures (22)

• Figure 2: Cross generalization across different data sources including StackExchange, Reddit, Quora, and HelpSteer2, where models trained on one source and predict preferences on the others. The values indicate test accuracy.
• Figure 3: A moment of epiphany occurs during WorldPM training on Qwen 72B, characterized by a sudden drop in loss and a corresponding spike in gradients.
• Figure 4: Comparison of test loss on subjective evaluation sets across annotation sources, with and without style control. HelpSteer2's expert annotations show minimal impact from style control, while crowdsourced annotations (ChatBot Arena) and AI annotations (GPT4) show substantial variations. The gap between controlled and uncontrolled conditions grows with training data and model size, reflecting WorldPM's reduced style preference.
• Figure 5: Analysis of alignment performance across WorldPM training scales. Both Alpaca Eval and Arena Hard implement distinct style control mechanisms to mitigate style preference in AI-based subjective evaluation. The figure demonstrates performance under both controlled and uncontrolled conditions. Arena Hard exhibits stable performance trends across control conditions, with 72B consistently superior to 7B and optimal performance achieved at larger training scales. However, Alpaca Eval shows substantial sensitivity to style control, with evaluation scores highly correlated with response length when style control is absent.
• Figure 6: Comparison of PM fine-tuning performance across different WorldPM training scales and baseline without WorldPM. Larger WorldPM scales demonstrate enhanced fine-tuning benefits.