MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe

TL;DR

This work tackles the efficiency bottlenecks of multimodal LLMs by introducing MiniCPM-V 4.5 (8B) with three core innovations: a unified 3D-Resampler for compact image/video encoding, a unified document knowledge and OCR learning paradigm, and a hybrid RL post-training strategy. The approach combines a lightweight visual encoder, cross-modal compression, and flexible reasoning modes to deliver strong vision-language performance at substantially reduced memory and inference cost. Empirical results on OpenCompass and VideoMME show competitive or superior performance against larger proprietary and open-source models, with notable efficiency gains such as 46.7% GPU memory cost and 8.7% inference time relative to a strong baseline, and as low as 9.9% inference time on VideoMME compared to prior SOTA. The paper also demonstrates robust training protocols, including unified pre-training data strategies, a dynamic document-OCR curriculum, and RLAIF-V, enabling reliable, long-horizon reasoning and reduced hallucinations, making the model appealing for scalable real-world use.

Abstract

Multimodal Large Language Models (MLLMs) are undergoing rapid progress and represent the frontier of AI development. However, their training and inference efficiency have emerged as a core bottleneck in making MLLMs more accessible and scalable. To address the challenges, we present MiniCPM-V 4.5, an 8B parameter model designed for high efficiency and strong performance. We introduce three core improvements in model architecture, data strategy and training method: a unified 3D-Resampler model architecture for highly compact encoding over images and videos, a unified learning paradigm for document knowledge and text recognition without heavy data engineering, and a hybrid reinforcement learning strategy for proficiency in both short and long reasoning modes. Comprehensive experimental results in OpenCompass evaluation show that MiniCPM-V 4.5 surpasses widely used proprietary models such as GPT-4o-latest, and significantly larger open-source models such as Qwen2.5-VL 72B. Notably, the strong performance is achieved with remarkable efficiency. For example, on the widely adopted VideoMME benchmark, MiniCPM-V 4.5 achieves state-of-the-art performance among models under 30B size, using just 46.7\% GPU memory cost and 8.7\% inference time of Qwen2.5-VL 7B.

Figures (13)

• Figure 1: An overview of the MiniCPM-V 4.5 architecture. The model processes diverse visual inputs, such as high-resolution images and high frame rate videos. After the image partitioning and video packing processes, these inputs are encoded by a visual encoder and then fed into the unified 3D-Resampler. This module efficiently compresses both image and video features into a compact token sequence (achieving up to 16$\times$ compression rate for images and an additional 6$\times$ for videos), which is then processed by the LLM decoder. The decoder can generate responses in two distinct styles: a concise, short reasoning mode or a step-by-step, long reasoning mode.
• Figure 2: Unified paradigm for document knowledge and OCR learning via dynamic visual corruption. We create a spectrum of training tasks through varied corruption levels: low corruption preserves readability to learn robust OCR, high corruption forces the model to perform contextual inference, and moderate corruption requires integrated inference from visual clues and context.
• Figure 3: Performance ablation of adding probability-based reward. We report OpenCompass scores, response length, and entropy on different training steps.
• Figure 4: A case of comprehensive real-world reasoning.
• Figure 5: A case of comprehensive real-world reasoning in Chinese.