The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective

TL;DR

This survey reframes MLLMs through a data-model co-development lens, arguing that data and models evolve in tandem rather than in isolation. It introduces a dual-axis taxonomy: data contributions to models and model contributions to data, organized along scaling and usability, then extends into synthesis and insights where models act as data creators, mappers, navigators, and extractors. The work systematically analyzes data acquisition, augmentation, diversity, condensation, mixture, packing, and cross-modal alignment, alongside prompts, demonstrations, and human-aligned data to boost instruction responsiveness, reasoning, and ethics, with comprehensive benchmarks for evaluation. It also outlines a future roadmap emphasizing infrastructures, externally-boosted and self-boosted development, and self-improving data loops to accelerate progress in multi-modal AI. Overall, the paper offers a structured framework and practical directions to harmonize data and model advances for scalable, safe, and capable MLLMs.

Abstract

The rapid development of large language models (LLMs) has been witnessed in recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the modality from text to a broader spectrum of domains, attracting widespread attention due to the broader range of application scenarios. As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is receiving increasingly widespread attention and recognition. Tracing and analyzing recent data-oriented works for MLLMs, we find that the development of models and data is not two separate paths but rather interconnected. On the one hand, vaster and higher-quality data contribute to better performance of MLLMs; on the other hand, MLLMs can facilitate the development of data. The co-development of multi-modal data and MLLMs requires a clear view of 1) at which development stages of MLLMs specific data-centric approaches can be employed to enhance certain MLLM capabilities, and 2) how MLLMs, utilizing those capabilities, can contribute to multi-modal data in specific roles. To promote the data-model co-development for MLLM community, we systematically review existing works related to MLLMs from the data-model co-development perspective. A regularly maintained project associated with this survey is accessible at https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.

Figures (6)

• Figure 1: The trends of cumulative numbers of papers on arXiv related to MLLMs and those related to both MLLMs and data, respectively.
• Figure 2: Taxonomy for MLLMs from the data-model co-development perspective and overview of §\ref{['sec-data-contrib-scaling']}-\ref{['sec-future']} with their inter-relationships. Data contributions for MLLMs (§\ref{['sec-data-contrib-scaling']} & \ref{['sec-data-contrib-usability']}) are organized in an objective-driven manner and ordered according to MLLM development stages, i.e., first scaling up for better performance then improving the usability. Model contributions for data (§\ref{['sec-model-contrib-synthesis']} & \ref{['sec-model-contrib-application']}) are organized by the roles played by models.
• Figure 3: Organization of data approaches tailored for scaling of MLLMs.
• Figure 4: Organization of data approaches tailored for the usability of MLLMs.
• Figure 5: Overview of the model contributions to multi-modal data in terms of data synthesis, categorized by the roles of models.

Theorems & Definitions (1)

• Definition 1