NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints

TL;DR

The paper tackles the practical challenge of building native end-to-end Multimodal Large Language Models (MLLMs) under data constraints and introduces NaViL, a native MoE-extended MLLM designed for cost-efficient training. It systematically analyzes architectural design choices—LLM initialization, mixture-of-experts, and visual encoder configurations—and derives scaling laws showing that LLM capacity drives gains while visual encoder benefits saturate unless scaled in tandem with the LLM. Three core findings emerge: pre-trained LLM initialization accelerates convergence and improves zero-shot capability; MoEs significantly boost training efficiency without increasing active parameters; and the optimal visual encoder size scales with the LLM size in a log-linear fashion, prompting joint scaling rather than fixed encoders. NaViL demonstrates competitive performance on 14 multimodal benchmarks with around 600M pretraining pairs and provides practical guidelines for future native MLLMs, highlighting the importance of end-to-end optimization and principled scaling for vision-language tasks.

Abstract

Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained LLMs through continuous multimodal pre-training. However, the multimodal scaling property of this paradigm remains difficult to explore due to the separated training. In this paper, we focus on the native training of MLLMs in an end-to-end manner and systematically study its design space and scaling property under a practical setting, i.e., data constraint. Through careful study of various choices in MLLM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, we further explore the scaling properties of the native MLLM and indicate the positively correlated scaling relationship between visual encoders and LLMs. Based on these findings, we propose a native MLLM called NaViL, combined with a simple and cost-effective recipe. Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs. Besides that, our findings and results provide in-depth insights for the future study of native MLLMs.

Figures (14)

• Figure 1: Comparison of design choices, scaling properties, and performance of our native MLLMs. We systematically investigate the designs and the scaling properties of native MLLMs under data constraints and yield valuable findings for building native MLLMs. After adopting these findings, our native MLLMs achieve competitive performance with top-tier MLLMs. $\mathcal{V}^*_{d,w}(\cdot)$ denotes the visual encoder with optimal parameter size.
• Figure 2: Effectiveness of LLM initialization. Left: The validation loss. The LLM initialized one converges much faster. Right: The zero-shot caption performance. Due to the lack of textual knowledge, the uninitialized model continues to lag behind.
• Figure 3: The validation loss of adding MoE or not. Using MoE extension will cause the loss to decrease more quickly.
• Figure 4: The validation loss and zero-shot caption performance of different visual encoders. The loss and performance only differ when the visual encoder is extremely wide or shallow.
• Figure 5: The validation loss when scaling up LLMs. With the same visual encoder (i.e.600M), the validation loss decreases log-linearly with the LLM size.