Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

Zicheng Kong, Dehua Ma, Zhenbo Xu, Alven Yang, Yiwei Ru, Haoran Wang, Zixuan Zhou, Fuqing Bie, Liuyu Xiang, Huijia Wu, Jian Zhao, Zhaofeng He

TL;DR

Omni-RRM introduces a fully open-source, rubric-grounded reward model that jointly handles text, image, video, and audio by generating structured, dimension-wise rationales and a final verdict. Central to the approach is Omni-Preference, a fully automated data engine that creates high-confidence, rubric-annotated preference pairs without human labeling, enabling a two-stage training regime: supervised fine-tuning to learn rubric-grounded outputs, followed by Group Relative Policy Optimization to sharpen discrimination on hard cases. Empirical results show state-of-the-art open-source results in video and audio benchmarks, strong image performance, and effective transfer to text-only preferences, with Best-of-N inference further validating its practical utility. The work demonstrates that rubric-grounded, omni-modal supervision provides a controllable, interpretable, and scalable path to aligning multimodal AI systems in real-world settings.

Abstract

Multimodal large language models (MLLMs) have shown remarkable capabilities, yet their performance is often capped by the coarse nature of existing alignment techniques. A critical bottleneck remains the lack of effective reward models (RMs): existing RMs are predominantly vision-centric, return opaque scalar scores, and rely on costly human annotations. We introduce \textbf{Omni-RRM}, the first open-source rubric-grounded reward model that produces structured, multi-dimension preference judgments with dimension-wise justifications across \textbf{text, image, video, and audio}. At the core of our approach is \textbf{Omni-Preference}, a large-scale dataset built via a fully automated pipeline: we synthesize candidate response pairs by contrasting models of different capabilities, and use strong teacher models to \emph{reconcile and filter} preferences while providing a modality-aware \emph{rubric-grounded rationale} for each pair. This eliminates the need for human-labeled training preferences. Omni-RRM is trained in two stages: supervised fine-tuning to learn the rubric-grounded outputs, followed by reinforcement learning (GRPO) to sharpen discrimination on difficult, low-contrast pairs. Comprehensive evaluations show that Omni-RRM achieves state-of-the-art accuracy on video (80.2\% on ShareGPT-V) and audio (66.8\% on Audio-HH-RLHF) benchmarks, and substantially outperforms existing open-source RMs on image tasks, with a 17.7\% absolute gain over its base model on overall accuracy. Omni-RRM also improves downstream performance via Best-of-$N$ selection and transfers to text-only preference benchmarks. Our data, code, and models are available at https://anonymous.4open.science/r/Omni-RRM-CC08.

Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

TL;DR

Omni-RRM introduces a fully open-source, rubric-grounded reward model that jointly handles text, image, video, and audio by generating structured, dimension-wise rationales and a final verdict. Central to the approach is Omni-Preference, a fully automated data engine that creates high-confidence, rubric-annotated preference pairs without human labeling, enabling a two-stage training regime: supervised fine-tuning to learn rubric-grounded outputs, followed by Group Relative Policy Optimization to sharpen discrimination on hard cases. Empirical results show state-of-the-art open-source results in video and audio benchmarks, strong image performance, and effective transfer to text-only preferences, with Best-of-N inference further validating its practical utility. The work demonstrates that rubric-grounded, omni-modal supervision provides a controllable, interpretable, and scalable path to aligning multimodal AI systems in real-world settings.

Abstract

Multimodal large language models (MLLMs) have shown remarkable capabilities, yet their performance is often capped by the coarse nature of existing alignment techniques. A critical bottleneck remains the lack of effective reward models (RMs): existing RMs are predominantly vision-centric, return opaque scalar scores, and rely on costly human annotations. We introduce \textbf{Omni-RRM}, the first open-source rubric-grounded reward model that produces structured, multi-dimension preference judgments with dimension-wise justifications across \textbf{text, image, video, and audio}. At the core of our approach is \textbf{Omni-Preference}, a large-scale dataset built via a fully automated pipeline: we synthesize candidate response pairs by contrasting models of different capabilities, and use strong teacher models to \emph{reconcile and filter} preferences while providing a modality-aware \emph{rubric-grounded rationale} for each pair. This eliminates the need for human-labeled training preferences. Omni-RRM is trained in two stages: supervised fine-tuning to learn the rubric-grounded outputs, followed by reinforcement learning (GRPO) to sharpen discrimination on difficult, low-contrast pairs. Comprehensive evaluations show that Omni-RRM achieves state-of-the-art accuracy on video (80.2\% on ShareGPT-V) and audio (66.8\% on Audio-HH-RLHF) benchmarks, and substantially outperforms existing open-source RMs on image tasks, with a 17.7\% absolute gain over its base model on overall accuracy. Omni-RRM also improves downstream performance via Best-of- selection and transfers to text-only preference benchmarks. Our data, code, and models are available at https://anonymous.4open.science/r/Omni-RRM-CC08.
Paper Structure (68 sections, 11 equations, 3 figures, 12 tables)

This paper contains 68 sections, 11 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modal Large Language Models
  4. Preference Alignment
  5. Rubric-grounded reward modeling
  6. Multimodal Reward Models
  7. Problem Formulation
  8. Method
  9. Omni-Preference: A Rubric-Grounded Dataset for Omni-Modal Reward Modeling
  10. Stage 1: Automated Candidate Pair Generation.
  11. Stage 2: Rubric Annotation, Reconciliation, and Filtering.
  12. Training Omni-RRM: A Progressive SFT--RL Approach
  13. Stage 1: Supervised Fine-Tuning (SFT).
  14. Stage 2: Reinforcement Learning with GRPO.
  15. Experiments
  16. ...and 53 more sections

Figures (3)

  • Figure 1: The overall pipeline for creating Omni-RRM. The process consists of two main phases. Top: Omni Preference Construction. Preference pairs are first automatically generated by contrasting outputs from a strong and a weak model. Powerful teacher models then annotate these pairs with rubric-grounded, multi-criteria justifications to create the final dataset. Bottom: Two-Stage Training. In Stage 1 (SFT), the policy model is trained on preference prompts using a standard NLL loss. In Stage 2 (GRPO), the model is further refined by generating multiple responses, scoring them with a rule-based reward function, and updating the policy based on the normalized reward.
  • Figure 2: Best-of-$N$ inference-time alignment with different judge models. We fix the base generator to Qwen2.5-Omni-7B and compare Greedy (single decoding), Self-consistency (majority vote over $N{=}5$ generations), and Best-of-$N$ (blue), which samples the same $N{=}5$ candidates and selects the final output via pairwise preference comparisons under a specified judge model (title of each subplot). We report accuracy (%) on MMMU (Image), Video-MME (Video), and AVQA (Audio), showing only the modalities supported by each judge. Values above the Best-of-$N$ bars denote absolute improvements over Self-consistency.
  • Figure 3: Modality-level ablations for omni-modal reward training (preference accuracy, %). All settings use the same 3B backbone and identical SFT+GRPO pipeline; only the training data composition differs. Rows correspond to training subsets (All / Drop Video / Drop Image / Audio-only) and columns correspond to evaluation benchmarks (VLReward / ShareGPT-Video / Audio-HH-RLHF).