Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models

TL;DR

Mixed-R1 proposes a unified reinforcement learning post-training framework for multimodal LLMs by constructing Mixed-45K, a diverse dataset, and a four-type reward design (plus a formatting bias) to jointly improve reasoning across tasks. A BMAS-based open-ended reward enables stable training without extra LLM judges, while integrating matching, chart, and IoU rewards addresses Yes/No, MCQ, and grounding tasks. Empirical results show consistent gains across multiple models and benchmarks, with ablations confirming the benefits of combining rewards and using 45K data. The approach provides a scalable, generalizable path toward unified reasoning improvements in multimodal LLMs.

Abstract

Recent works on large language models (LLMs) have successfully demonstrated the emergence of reasoning capabilities via reinforcement learning (RL). Although recent efforts leverage group relative policy optimization (GRPO) for MLLMs post-training, they constantly explore one specific aspect, such as grounding tasks, math problems, or chart analysis. There are no works that can leverage multi-source MLLM tasks for stable reinforcement learning. In this work, we present a unified perspective to solve this problem. We present Mixed-R1, a unified yet straightforward framework that contains a mixed reward function design (Mixed-Reward) and a mixed post-training dataset (Mixed-45K). We first design a data engine to select high-quality examples to build the Mixed-45K post-training dataset. Then, we present a Mixed-Reward design, which contains various reward functions for various MLLM tasks. In particular, it has four different reward functions: matching reward for binary answer or multiple-choice problems, chart reward for chart-aware datasets, IoU reward for grounding problems, and open-ended reward for long-form text responses such as caption datasets. To handle the various long-form text content, we propose a new open-ended reward named Bidirectional Max-Average Similarity (BMAS) by leveraging tokenizer embedding matching between the generated response and the ground truth. Extensive experiments show the effectiveness of our proposed method on various MLLMs, including Qwen2.5-VL and Intern-VL on various sizes. Our dataset and model are available at https://github.com/xushilin1/mixed-r1.

Figures (5)

• Figure 1: Results of Our Mixed-R1. Our post-training method improves Qwen2.5-VL on four different challenging benchmarks by large margins, on both the 3B model and the 7B model.
• Figure 2: The distribution of the training dataset (a) and dataset filter pipeline (b).
• Figure 3: Mixed-45K data types and reward examples used in our Mixed-45K.
• Figure 4: Comparison of different Open-Ended reward designs on Qwen2.5-VL-3B.
• Figure 5: Comparison examples of our Mixed-R1 compared with Qwen2.5-VL-7B-Instruct. We highlight the related reasoning words in orange.