Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, Oleksii Kuchaiev

TL;DR

HelpSteer3-Preference delivers a large, openly licensed human-annotated preference dataset spanning STEM, Code, and multilingual tasks, addressing the need for diverse, high-quality RLHF data. The authors demonstrate strong gains by training both Bradley–Terry and Generative Reward Models on RM-Bench and JudgeBench, and show that these rewards can effectively guide RLHF-aligned policies via RLOO. The dataset construction emphasizes specialist annotators, rigorous quality control, and careful post-processing, yielding high inter-rater reliability and low position bias. Overall, the work provides a scalable, open benchmark and demonstrates practical benefits for reward modeling and policy alignment in multilingual and code-rich settings, while discussing limitations and future directions for broader coverage and modalities.

Abstract

Preference datasets are essential for training general-domain, instruction-following language models with Reinforcement Learning from Human Feedback (RLHF). Each subsequent data release raises expectations for future data collection, meaning there is a constant need to advance the quality and diversity of openly available preference data. To address this need, we introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0), high-quality, human-annotated preference dataset comprising of over 40,000 samples. These samples span diverse real-world applications of large language models (LLMs), including tasks relating to STEM, coding and multilingual scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This represents a substantial improvement (~10% absolute) over the previously best-reported results from existing RMs. We demonstrate HelpSteer3-Preference can also be applied to train Generative RMs and how policy models can be aligned with RLHF using our RMs. Dataset (CC-BY-4.0): https://huggingface.co/datasets/nvidia/HelpSteer3#preference Models (NVIDIA Open Model): https://huggingface.co/collections/nvidia/reward-models-68377c5955575f71fcc7a2a3

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

TL;DR

HelpSteer3-Preference delivers a large, openly licensed human-annotated preference dataset spanning STEM, Code, and multilingual tasks, addressing the need for diverse, high-quality RLHF data. The authors demonstrate strong gains by training both Bradley–Terry and Generative Reward Models on RM-Bench and JudgeBench, and show that these rewards can effectively guide RLHF-aligned policies via RLOO. The dataset construction emphasizes specialist annotators, rigorous quality control, and careful post-processing, yielding high inter-rater reliability and low position bias. Overall, the work provides a scalable, open benchmark and demonstrates practical benefits for reward modeling and policy alignment in multilingual and code-rich settings, while discussing limitations and future directions for broader coverage and modalities.

Abstract

Preference datasets are essential for training general-domain, instruction-following language models with Reinforcement Learning from Human Feedback (RLHF). Each subsequent data release raises expectations for future data collection, meaning there is a constant need to advance the quality and diversity of openly available preference data. To address this need, we introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0), high-quality, human-annotated preference dataset comprising of over 40,000 samples. These samples span diverse real-world applications of large language models (LLMs), including tasks relating to STEM, coding and multilingual scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This represents a substantial improvement (~10% absolute) over the previously best-reported results from existing RMs. We demonstrate HelpSteer3-Preference can also be applied to train Generative RMs and how policy models can be aligned with RLHF using our RMs. Dataset (CC-BY-4.0): https://huggingface.co/datasets/nvidia/HelpSteer3#preference Models (NVIDIA Open Model): https://huggingface.co/collections/nvidia/reward-models-68377c5955575f71fcc7a2a3
Paper Structure (62 sections, 1 equation, 2 figures, 11 tables)

This paper contains 62 sections, 1 equation, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Dataset
  3. Dataset Construction
  4. Prompt Collection
  5. Response Generation
  6. Multi-turn Prompt In-filling
  7. Preference Annotation
  8. Preference Post-Processing
  9. Dataset Analysis
  10. Improvements over HelpSteer2-Preference
  11. Diverse Programming and Natural Languages
  12. High Inter-Rater Reliability
  13. Low Position Bias
  14. Preference Distributions
  15. Reward Models
  16. ...and 47 more sections

Figures (2)

  • Figure 1: Win rate by setting for English and Multilingual RM. $y_c$ refers to the chosen response while $y_r$ refers to the rejected response. $y^{\emptyset}$ refers to the concise response, $y^{L}$ refers to the verbose response while $y^{L,M}$ refers to the verbose response with markdown formatting following liu2025rmbench.
  • Figure 2: Distribution of overall preferences in HelpSteer3-Preference (HS3-Pref) subsets in comparison with HelpSteer2-Preference (HS2-Pref). $A>>>B$ means Response 1 is much better than Response 2, $A>>B$ means better, $A>B$ means slightly better (and vice-versa).