Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preference Optimization for Review Question Generation Improves Writing Quality

Karun Sharma, Vidushee Vats, Shengzhi Li, Yuxiang Wang, Zhongtian Sun, Prayag Tiwari

TL;DR

IntelliReward is developed, a novel reward model built from a frozen autoregressive LLM with trainable multi-head transformers over the final 50 token states which outperforms API-based SFT baselines in predicting expert-level human preferences and shows consistent improvements on reasoning and writing benchmarks.

Abstract

Peer review relies on substantive, evidence-based questions, yet existing LLM-based approaches often generate surface-level queries, drawing over 50\% of their question tokens from a paper's first page. To bridge this gap, we develop IntelliReward, a novel reward model built from a frozen autoregressive LLM with trainable multi-head transformers over the final 50 token states, which outperforms API-based SFT baselines in predicting expert-level human preferences. By applying Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) with IntelliReward, we train IntelliAsk, a question-generation model aligned with human standards of effort, evidence, and grounding. We find consistent improvements on reasoning and writing benchmarks, suggesting reviewer-question quality correlates with broader capabilities. Compared to the Qwen3-32B base model, IntelliAsk shows measurable gains across diverse benchmarks, specifically improving performance on reasoning tasks like MuSR (68.3 vs 64.7 Acc) and complex writing evaluations such as WritingBench (8.31 vs 8.07). We release our implementation, expert preference annotations, and the IntelliReward model to provide an automatic evaluation benchmark for grounding, effort, and evidence in LLM-generated review questions.

Preference Optimization for Review Question Generation Improves Writing Quality

TL;DR

IntelliReward is developed, a novel reward model built from a frozen autoregressive LLM with trainable multi-head transformers over the final 50 token states which outperforms API-based SFT baselines in predicting expert-level human preferences and shows consistent improvements on reasoning and writing benchmarks.

Abstract

Peer review relies on substantive, evidence-based questions, yet existing LLM-based approaches often generate surface-level queries, drawing over 50\% of their question tokens from a paper's first page. To bridge this gap, we develop IntelliReward, a novel reward model built from a frozen autoregressive LLM with trainable multi-head transformers over the final 50 token states, which outperforms API-based SFT baselines in predicting expert-level human preferences. By applying Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) with IntelliReward, we train IntelliAsk, a question-generation model aligned with human standards of effort, evidence, and grounding. We find consistent improvements on reasoning and writing benchmarks, suggesting reviewer-question quality correlates with broader capabilities. Compared to the Qwen3-32B base model, IntelliAsk shows measurable gains across diverse benchmarks, specifically improving performance on reasoning tasks like MuSR (68.3 vs 64.7 Acc) and complex writing evaluations such as WritingBench (8.31 vs 8.07). We release our implementation, expert preference annotations, and the IntelliReward model to provide an automatic evaluation benchmark for grounding, effort, and evidence in LLM-generated review questions.
Paper Structure (43 sections, 3 equations, 13 figures, 11 tables)

This paper contains 43 sections, 3 equations, 13 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Question Extraction and Curation
  3. Large-Scale Extraction of Questions from Openreview Reviews
  4. Benchmarking SOTA Reasoning LLMs Against Humans
  5. Human Preference and Annotation Study
  6. Rubrics For Assessing Question Quality: Effort, Evidence, and Grounding
  7. Analysis of Human vs. Model-Generated Questions
  8. SFT on Filtered Human Questions
  9. Training IntelliAsk: A specialized model for asking critical questions
  10. Reward Model : IntelliReward
  11. Reward Model Architecture and Training
  12. Reward Model Architecture.
  13. Training Objective and Inference
  14. Reward Model Training.
  15. RL using IntelliReward Reward Model
  16. ...and 28 more sections

Figures (13)

  • Figure 1: Architecture and training of the IntelliReward.
  • Figure 2: Waterfall diagram illustrating progressive instance filtering at each stage of the data curation process.
  • Figure 3: The figure show the difference in reward curves for Qwen2.5-7B (SFT) and IntelliAsk during training.
  • Figure 4: Distribution of question lengths across sources. Kernel density estimates show that human-authored questions exhibit the highest variance, reflecting greater diversity. Qwen2.5-32B produces the shortest questions, while Gemini 2.5 Pro generates the longest.
  • Figure 5: The figures show the distribution of votes on Effort, Evidence and Factual metrics for various sources of questions.
  • ...and 8 more figures