OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generation

TL;DR

OpenGenAlign addresses the gap in reward-model data for open-ended, long-context generation by introducing a 33K training-pair dataset, 9K development samples, and a 1.5K held-out benchmark across QA, Data-to-Text, and Summarization. It trains a Bradley-Terry reward model on this data and demonstrates that PPO-based policy optimization yields improved long-context generation, with the reward model aligning with human judgments and outperforming baselines. The work also shows that OpenGenAlign enables effective guided generation on out-of-distribution tasks and can be integrated with other preference datasets to build more versatile reward models. The dataset is slated for public release to advance research and practical applications in long-context generation and RLHF pipelines.

Abstract

Reward Modeling is critical in evaluating and improving the generation of Large Language Models (LLMs). While numerous recent works have shown its feasibility in improving safety, helpfulness, reasoning, and instruction-following ability, its capability and generalization to open-ended long-context generation is still rarely explored. In this paper, we introduce OpenGenAlign, a framework and a high-quality dataset designed to develop reward models to evaluate and improve hallucination-free, comprehensive, reliable, and efficient open-ended long-context generation. We define four key metrics to assess generation quality and develop an automated pipeline to evaluate the outputs of multiple LLMs across long-context QA, Data-to-Text, and Summarization scenarios using o3, ending up with 33K high-quality preference data with a human agreement rate of 81\%. Experimental results first demonstrate that existing reward models perform suboptimally on the held-out benchmark. And Our trained reward model achieves superior performance in the benchmark and effectively improves the generation quality of the policy models using Reinforcement Learning (RL). Additionally, OpenGenAlign could be used for effective guided generation in existing datasets. Furthermore, we demonstrate that the OpenGenAlign could be integrated with reward data from other domains to achieve better performance.

Figures (6)

• Figure 1: Overview of our data labeling method and our experiments based on it in the open-ended long-context Scenario. We use o3 as the judge to evaluate the quality of the generation from multiple models. We then train the reward models and use them for Reinforcement Learning.
• Figure 2: Illustration of our data annotation method. Given a sample and two responses, we prompt o3 to provide judgments based on each metric and aggregate the results to construct pairs.
• Figure 3: Statistics of the preferred rate for each model during preference pairs construction phase.
• Figure 4: WebGLM System Prompt
• Figure 5: XSum System Prompt