AutoRubric-R1V: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning

TL;DR

AutoRubric-R1V addresses spurious multimodal reasoning that arises when RLVR only rewards final-correct answers. It automatically generates problem-specific rubrics by aggregating consistent steps across successful trajectories and uses these rubrics to guide LLM-based trajectory evaluation, combining rubric rewards with outcome rewards within a GRPO framework. The method achieves state-of-the-art results on six multimodal reasoning benchmarks and demonstrates substantially improved reasoning faithfulness, supported by dedicated faithfulness evaluations and ablations. By eliminating the need for human annotations or stronger teacher models, AutoRubric-R1V offers a data-efficient, robust approach to faithful multimodal reasoning with publicly releasable rubric data and code.

Abstract

Multimodal large language models (MLLMs) have rapidly advanced from perception tasks to complex multi-step reasoning, yet reinforcement learning with verifiable rewards (RLVR) often leads to spurious reasoning since only the final-answer correctness is rewarded. To address this limitation, we propose AutoRubric-R1V, a framework that integrates RLVR with process-level supervision through automatically collected rubric-based generative rewards. Our key innovation lies in a scalable self-aggregation method that distills consistent reasoning checkpoints from successful trajectories, enabling problem-specific rubric construction without human annotation or stronger teacher models. By jointly leveraging rubric-based and outcome rewards, AutoRubric-R1V achieves state-of-the-art performance on six multimodal reasoning benchmarks and substantially improves reasoning faithfulness in dedicated evaluations.

Figures (11)

• Figure 1: Illustration of a multimodal reasoning question together with two model-generated reasoning traces that both reach the correct answer. Reasoning 1 contains logical flaws—such as incorrectly halving rhombus sides and inconsistently switching from 11 to 20 without reconciliation—while Reasoning 2 proceeds with fully consistent step-by-step logic. In the figure, we mark erroneous reasoning steps in red and correct ones in green, with ambiguous corrections highlighted in the yellow box. Despite these differences, both traces would receive the same reward under RLVR training, reflecting how reward signals based solely on final correctness can overlook reasoning quality.
• Figure 2: Our framework extends GRPO with rubric-based reasoning rewards. Top: Rubrics Construction. We automatically generate rubrics by comparing multiple reasoning trajectories of the same question that reach the correct answer. Common steps across multiple correct trajectories are preserved as necessary reasoning, while rare or inconsistent steps are discarded as likely errors or irrelevant. This yields a set of high-quality rubrics without relying on stronger proprietary MLLMs. Bottom: Training Process. In vanilla GRPO, answer-based rewards assign identical scores to all trajectories with the same final answer, failing to capture differences in reasoning quality. By integrating rubric-based scoring, trajectories are further distinguished at the step level, and the combined rubric and answer rewards produce differentiated advantages that better guide the policy model toward logically consistent reasoning paths.
• Figure 3: First row: Comparison of AutoRubric-R1V and vanilla GRPO in terms of training dynamics (answer reward, rubric-based reasoning reward, and response length of rollouts). Second row: Evaluation accuracy of different training steps on three multimodal mathematical benchmarks.
• Figure 4: Illustration of a problem with the constructed rubrics, two reasoning trajectories produced by the policy model, and the rubric-based reasoning rewards assigned by the judge model.
• Figure 5: The prompt for rubric construction.