Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training

Ran Xu, Tianci Liu, Zihan Dong, Tony You, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, Haoyu Wang

TL;DR

Rubric-ARM addresses the challenge of evaluating non-verifiable, open-ended LLM outputs by jointly learning a rubric generator and a rubric-conditioned judge through alternating reinforcement learning. The authors formalize rubrics as latent actions and provide a theoretical variance analysis showing how the alternating schedule stabilizes learning. Empirically, Rubric-ARM achieves state-of-the-art performance on a wide set of reward-model benchmarks and delivers consistent improvements for downstream policy alignment in both offline and online RL settings. The work demonstrates that interpretable, rubric-grounded signals can yield superior, generalizable alignment signals and offers practical tools for scalable rubric generation and evaluation.

Abstract

Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training

TL;DR

Rubric-ARM addresses the challenge of evaluating non-verifiable, open-ended LLM outputs by jointly learning a rubric generator and a rubric-conditioned judge through alternating reinforcement learning. The authors formalize rubrics as latent actions and provide a theoretical variance analysis showing how the alternating schedule stabilizes learning. Empirically, Rubric-ARM achieves state-of-the-art performance on a wide set of reward-model benchmarks and delivers consistent improvements for downstream policy alignment in both offline and online RL settings. The work demonstrates that interpretable, rubric-grounded signals can yield superior, generalizable alignment signals and offers practical tools for scalable rubric generation and evaluation.

Abstract

Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
Paper Structure (37 sections, 4 theorems, 33 equations, 6 figures, 14 tables)

This paper contains 37 sections, 4 theorems, 33 equations, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Rubrics.
  5. Learning Objective.
  6. Rubric-ARM: Alternating RL for Rubric Generation and Judging
  7. Stage I: SFT Warmup
  8. Stage II: Alternating Reinforcement Learning
  9. (i) RL for Judge $\pi_j$ with the current $\pi_{r}$.
  10. (ii) RL for Rubric Generator $\pi_r$ with the current $\pi_{j}$.
  11. Connection to EM Algorithm.
  12. Policy Model Post-training with Rubric-ARM
  13. Theoretical Analysis
  14. Variance Decomposition
  15. Variance Domination in Early Training
  16. ...and 22 more sections

Key Result

Proposition 5.1

Conditioned on a reused rubric $\bar{r}$, the variance of the judge's gradient estimator $\widehat{g}_A$ is solely determined by the judge's binary classification uncertainty:

Figures (6)

  • Figure 1: The overall framework for Rubric-ARM.
  • Figure 2: Performance of different judge and reward models on WritingPreferenceBench.
  • Figure 3: Comparison of trained policy models on IFBench. Results of baselines except Rubric-RM (IterDPO) are from OpenRubrics liu2025openrubrics.
  • Figure 4: Comparison of trained policy models on Create Writing Benchmark v3. Results of baselines except Rubric-RM are from RuscaRL zhou2025breaking.
  • Figure 5: Performance of iterative DPO with Rubric-ARM across three iterations.
  • ...and 1 more figures

Theorems & Definitions (9)

  • Proposition 5.1: Judge Variance under Strategy A
  • Proposition 5.2: Generator Variance under Strategy B
  • Remark 5.4
  • Theorem 5.5: Strict Variance Domination
  • Remark 5.6: Implication for Training Stability
  • Lemma B.1
  • proof
  • proof
  • proof