Towards Efficient Multimodal Unified Reasoning Model via Model Merging
Qixiang Yin, Huanjin Yao, Jianghao Chen, Jiaxing Huang, Zhicheng Zhao, Fei Su
TL;DR
Tiny-R1V introduces a lightweight 3B multimodal model trained via a two-stage framework to achieve efficient, unified reasoning across math, charts, tables, and OCR. Stage 1 employs Length-Informed Relative Policy Optimization (LIPO) to favor concise yet accurate reasoning, reducing token usage while maintaining quality. Stage 2 uses Adaptive Model Merging (AMM) with dual weights and gradient projection regularization to integrate multiple specialist models into a single architecture without additional data. Across ten benchmarks, Tiny-R1V achieves strong cross-task performance, outperforming several merging baselines and competing with larger models while delivering substantial inference efficiency gains. The combination of LIPO and AMM provides a practical, data-efficient approach to scalable, multimodal reasoning in lightweight models.
Abstract
Although Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across diverse tasks, they encounter challenges in terms of reasoning efficiency, large model size and overthinking. However, existing lightweight MLLMs lack the capability to balance high efficiency and performance at a small scale. To this end, we propose Tiny-R1V, a novel lightweight 3B model that achieves faster inference and higher accuracy via a two-stage optimization, while unifying multimodal reasoning across multiple tasks with fewer inference tokens. In the first stage, Tiny-R1V introduces Length-Informed Relative Policy Optimization (LIPO), a new reinforcement learning method, to train each reasoning model, including mathematical reasoning, chart reasoning, and OCR capability. The LIPO dynamically adjusts the advantages of responses within groups by prioritizing concise yet high-quality responses to encourage the generation of shorter and more accurate responses. In the second stage, we propose Adaptive Model Merging (AMM), a training-free model merging method that merges multiple specialist models into a unified architecture. Specifically, AMM adaptively adjusts the weights of task vectors via a novel gradient projection regularization loss function, thus mitigating redundant conflicts between them. Extensive evaluations on ten widely-used reasoning benchmarks covering mathematics, structured data (charts, tables, documents), OCR, and general capabilities showcase the superior performance of Tiny-R1V, enabling lightweight models to excel in diverse multimodal reasoning tasks. Code will be available at \href{https://github.com/buptyqx/Tiny-R1V}{https://github.com/buptyqx/Tiny-R1V}