Online Rubrics Elicitation from Pairwise Comparisons

TL;DR

Static rubrics for reward modeling can miss emergent desiderata and be vulnerable to reward hacking in long-horizon open-ended tasks. OnlineRubrics introduces online elicitation of criteria from pairwise comparisons between current and reference policy responses, using an LLM-based extractor to augment rubrics during GRPO-based training. Empirically, it yields up to 8 percentage-point gains over training with static rubrics across generalist and expert benchmarks and provides qualitative insights into the evolving rubric themes such as grounding, practicality, and organization. This online, adaptive approach improves robustness and coverage of evaluation criteria, providing a practical path toward more resilient alignment for open-ended generation tasks.

Abstract

Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and reasoning.

Figures (13)

• Figure 1: At any step during training, OnlineRubrics starts off by considering a pair of responses, one of which is from the current policy before updates and another from a control model e.g. reference model. We follow with LLM-based rubrics elicitation and deduplication steps to generate a set of elicited criteria. These criteria along with existing criteria (e.g. human-written or synthetic) are used to create the reward in the policy gradient algorithm.
• Figure 2: Abbreviated system prompt template used for eliciting new criteria from pairwise response comparisons, see full prompt in \ref{['fig:extractor_prompt_full']}.
• Figure 3: Performance of different LLM graders. AUC score is calculated using the receiver operating characteristic (ROC) curve. The best grader is the one with the highest AUC score and the lowest inference cost per sample. Models on the Pareto frontier (shown as a red dotted line) are the best trade-off between the two metrics. We choose GPT-4.1-mini as our default grader, balancing alignment quality with inference cost.
• Figure 4: Results on the evaluation set of the Generalist and Expert datasets during training (higher is better). The evaluation set is fixed and does not contain any elicited rubrics. Both OnlineRubrics methods outperform using Offline Rubrics (Human) or LLM-judge Score (a Likert scale).
• Figure 5: Data sample from the Generalist Rubrics dataset.

Theorems & Definitions (2)

• Proposition 1
• proof