Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

TL;DR

This survey tackles the problem of data mixture for large language model training under fixed compute, focusing on how to assign domain weights across multiple data sources to maximize downstream performance. It introduces a fine-grained offline/online taxonomy, aligning offline methods with heuristic, algorithm-based, and function-fitting categories and mapping algorithm-based methods to min-max and bi-level optimization while treating online approaches as dynamic variants of offline strategies. The paper systematically summarizes problem formulations, representative algorithms, and the trade-offs of each method, and discusses practical challenges such as transferability across models, scalability with domain count, and the relationship between loss surrogates and actual task performance. By connecting data-mixing methods to optimization paradigms and scaling laws, it provides a unified perspective that informs both theoretical understanding and practical design for data-efficient LLM pretraining and fine-tuning.

Abstract

Training large language models with data collected from various domains can improve their performance on downstream tasks. However, given a fixed training budget, the sampling proportions of these different domains significantly impact the model's performance. How can we determine the domain weights across different data domains to train the best-performing model within constrained computational resources? In this paper, we provide a comprehensive overview of existing data mixture methods. First, we propose a fine-grained categorization of existing methods, extending beyond the previous offline and online classification. Offline methods are further grouped into heuristic-based, algorithm-based, and function fitting-based methods. For online methods, we categorize them into three groups: online min-max optimization, online mixing law, and other approaches by drawing connections with the optimization frameworks underlying offline methods. Second, we summarize the problem formulations, representative algorithms for each subtype of offline and online methods, and clarify the relationships and distinctions among them. Finally, we discuss the advantages and disadvantages of each method and highlight key challenges in the field of data mixture.