Reward Modeling for Scientific Writing Evaluation

TL;DR

This work introduces SciRM and SciRM-Ref, cost-efficient reward models tailored for evaluating scientific writing across multiple aspects. By conditioning on explicit constitutions and employing a two-stage GRPO-based training regime, the approach improves domain-grounded reasoning and generalizes to unseen tasks without task-specific retraining. Extensive experiments across related-work and RevUtil tasks, plus unseen-aspect and unseen-task evaluations, show robust improvements over baselines and strong generalization, especially for reasoning-intensive evaluation. The study delivers open-source evaluators capable of providing fine-grained, aspect-level feedback, paving the way for more reliable, interpretable, and reusable scientific writing evaluation in real-world workflows.

Abstract

Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage the domain knowledge to satisfy the task specifications. While scientific text generation has been widely studied, its evaluation remains a challenging and open problem. It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks while adhering to their distinct requirements. However, existing LLM-based judges and reward models are primarily optimized for general-purpose benchmarks with fixed scoring rubrics and evaluation criteria. Consequently, they often fail to reason over sparse knowledge of scientific domains when interpreting task-dependent and multi-faceted criteria. Moreover, fine-tuning for each individual task is costly and impractical for low-resource settings. To bridge these gaps, we propose cost-efficient, open-source reward models tailored for scientific writing evaluation. We introduce a two-stage training framework that initially optimizes scientific evaluation preferences and then refines reasoning capabilities. Our multi-aspect evaluation design and joint training across diverse tasks enable fine-grained assessment and robustness to dynamic criteria and scoring rubrics. Experimental analysis shows that our training regime strongly improves LLM-based scientific writing evaluation. Our models generalize effectively across tasks and to previously unseen scientific writing evaluation settings, allowing a single trained evaluator to be reused without task-specific retraining.

Figures (30)

• Figure 1: Example demonstration of how we formalize scientific writing evaluation and the outputs of the review utility evaluation task. Vanilla LLM-based judges fail to properly reason over the task-specific evaluation criteria and provided examples. Contradictory statements are highlighted in different colors. In contrast, our SciRM model successfully incorporates the given criteria and examples into its reasoning process and correctly evaluates the scientific artifact.
• Figure 2: Overview of SciRM and SciRM-Ref training and testing pipeline. Diverse scientific artifacts are used to construct training data with multiple evaluation aspects and scoring rubrics (see Section \ref{['sec:dataset']} for details). Models are trained via GRPO in two stages to optimize task specifications and reasoning capabilities, and are evaluated on both seen and unseen scientific writing evaluation tasks.
• Figure 3: Review utility evaluation results. Results to the left of the vertical dashed lines correspond to the baseline models, whereas results to the right correspond to our models.
• Figure 4: Related work evaluation results. Results to the left of the vertical dashed lines correspond to the baseline models, whereas results to the right correspond to our models.
• Figure 5: System prompt that is commonly used across each evaluation task and aspect.