Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training

TL;DR

Paper investigates how text-only pre-training leads to visual priors in LLMs, revealing a decomposition into separable perception and reasoning priors with distinct data origins and scaling. It demonstrates that reasoning priors emerge from reasoning-centric data and transfer across modalities, while perception priors arise from broad corpora and depend on the vision encoder and vision-tuning data; a data-centric recipe is proposed and validated at 1T token scale. The work introduces the Multi-Level Existence Bench (MLE-Bench) to assess perception across object sizes and shows that an optimal data mixture (about 60% reasoning and 15% visual content) maximizes vision-enabled tasks while preserving language performance. Scaling experiments with 7B models further validate that a balanced data mixture yields strong multimodal performance, underscoring a practical pathway to deliberate cultivation of visual priors in multimodal LLMs.

Abstract

Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, and in some cases, to perform visual tasks without ever having seen an image. Through systematic analysis, we reveal that visual priors-the implicit, emergent knowledge about the visual world acquired during language pre-training-are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that an LLM's latent visual reasoning ability is predominantly developed by pre-training on reasoning-centric data (e.g., code, math, academia) and scales progressively. This reasoning prior acquired from language pre-training is transferable and universally applicable to visual reasoning. In contrast, a perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose a data-centric recipe for pre-training vision-aware LLMs and verify it in 1T token scale pre-training. Our findings are grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MLLM construction pipeline-from LLM pre-training to visual alignment and supervised multimodal fine-tuning-across five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we propose and investigate several hypotheses, and introduce the Multi-Level Existence Bench (MLE-Bench). Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.

Figures (13)

• Figure 1: Impact of model and data sizes. The plots illustrate the performance of MLLMs, built upon LLMs of five different sizes (340M to 13B parameters), as a function of the amount of web-crawl pre-training data (0B to 100B tokens). The general trend shows that performance improves with both increasing model size and data volume, but the scaling behavior differs across task categories.
• Figure 2: Impact of pre-training data sources. The bar charts illustrate the downstream VQA performance of MLLMs built upon a 3B parameter LLM, where each LLM was pre-trained on 30B tokens from a single, specific data source. The plots show that performance varies significantly depending on the pre-training sources.
• Figure 3: Impact of reasoning-centric and visual data categories and proportions. The plots illustrate how varying the proportion of specific data categories in the pre-training mix affects downstream VQA performance. Left plot (Reasoning-centric data): The plot shows that increasing the share of reasoning-centric data leads to progressive and significant performance gains, with benefits scaling up to a 75% proportion before plateauing. This indicates that a strong reasoning foundation is critical for enhancing visual abilities. Right plot (Visual world data): In contrast, the right plot, showing data that explicitly describes the visual world demonstrates rapidly diminishing returns. Only a small amount of this data is crucial to establish a baseline.
• Figure 4: Correlation matrix for VQA performances. The matrix reveals two loosely-coupled skill clusters: one axis for perception (General/OCR) and another for reasoning (Knowledge/Vision-Centric).
• Figure 5: Universality of the learned visual priors. The plots show the VQA performance of MLLMs built using three distinct vision encoders based on the proportion of reasoning-centric data used in the LLM's pre-training mix. Despite differences in their absolute performance, all three configurations show a consistent improvement on reasoning-heavy tasks as the LLM's reasoning pre-training proportion increases, similar to trends observed before, demonstrating the universality of the reasoning prior.