CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

TL;DR

CARMO (Context-Aware Reward Modeling) proposes dynamically generated, context-specific evaluation criteria to ground reward models and mitigate reward hacking in RLHF. The framework uses a two-stage process: an LLM generates task-tailored criteria, then these criteria guide absolute or relative response evaluation, with a knowledge-distillation path to transfer capabilities to smaller open-source models. Theoretical analysis formalizes why adaptive criteria outperform fixed rubrics under distribution shifts and spurious correlations, supported by experiments across multiple benchmarks showing state-of-the-art zero-shot RewardBench performance and strong alignment gains on Mistral-Base and related models. CARMO also demonstrates practical benefits by enabling robust preference data generation for RLHF methods like DPO and SWEPO, and releasing open-source datasets to foster reproducibility and broader adoption.

Abstract

Reward modeling in large language models is susceptible to reward hacking, causing models to latch onto superficial features such as the tendency to generate lists or unnecessarily long responses. In reinforcement learning from human feedback (RLHF) and more generally during post-training flawed reward signals often lead to outputs that optimize for these spurious correlates instead of genuine quality or correctness. We propose Context-Aware Reward Modeling (CARMO), a novel approach that first generates dynamic, context-relevant criteria to ground the reward model before producing reward scores. Unlike prior methods that rely on static rubrics, CARMO leverages large language models (LLMs) to adaptively create evaluation criteria such as logical consistency, clarity, and depth tailored to the user query. Our theoretical analysis shows that such criteria generation can mitigate reward hacking. We further demonstrate that CARMO can be distilled into smaller models, reducing the computational cost of alignment. We establish a new state-of-the-art performance in zero-shot settings for generative models, achieving a 2.1\% improvement on Reward Bench. Furthermore, alignment performed on the CARMO-curated preference dataset achieves 22.5\% and 21.1\% LC-WR and WR, respectively, on Mistral-Base (7B).

Figures (6)

• Figure 1: Our paper improves scoring for (Q,A) pairs from generative models via dynamic criteria generation. Naive methods either directly ask for response score, or use a fixed external criteria. We propose two variants -- Carmo single-pass method with dynamic criteria generation, and Carmo two-pass method separating criteria generation from feedback and scoring.
• Figure 2: Training pipeline for fine-tuning small models for criteria generation as well as query feedback and scoring.
• Figure 3: System architecture for training an aligned student LLM using preference data from a large language model that uses $\textsc{Carmo}$ rating Algorithm.
• Figure 4: Performance analysis of single-stage \ref{['subsec:carmo_single_stage_prompt']} and two-stage \ref{['subsec:carmo_two_stage_prompt']} prompt setting of $\textsc{Carmo}$ on Reward Bench for gpt-4o.
• Figure 5: Performance analysis of default \ref{['subsubsec:carmo_two_stage_normal_prompt']} and detailed \ref{['subsubsec:carmo_two_stage_detailed_prompt']} prompt setting of $\textsc{Carmo}$ on Reward Bench for gpt-4o.

Theorems & Definitions (28)

• Theorem 1: A model using relevant features outperforms one using spurious features
• Theorem 2: Failure of a Fixed Finite Rubric
• Remark 1: Approximate Independence
• Lemma 1: Spurious Single‐Dimension Predictors
• proof
• Remark 2: Finite Data
• Lemma 2: Relevant Single‐Dimension Predictors
• proof
• Theorem 3: Spurious Single‐Axis vs. Relevant Single‐Axis
• proof