BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

TL;DR

BLISS tackles data selection for pretraining large language models by introducing a self-contained bilevel framework that uses a lightweight proxy model and a score model to estimate the long-term influence of training samples, enabling efficient data curation without external oracle models. The outer objective trains the score model to optimize downstream validation performance, while the inner objective trains the proxy on weighted data with KL alignment to the target LLM, simulating long-run effects. Empirically, BLISS outperforms state-of-the-art baselines across multiple model scales (e.g., 410M, 1B, and 2.8B) and architectures (Pythia and LLaMA), achieving faster convergence and better generalization, including a reported 1.7x speedup in the 1B setting. The approach reduces reliance on proprietary data or expensive oracle feedback, offering practical gains in data efficiency for large-scale pretraining and demonstrating robustness across tasks and architectures. Overall, BLISS provides a principled, scalable path for effective data selection in scratch-trained LLM pretraining with broad applicability to future multi-architecture regimes.

Abstract

Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks. However, existing approaches often require leveraging external pretrained models, making it difficult to disentangle the effects of data selection from those of the external pretrained models. In addition, they often overlook the long-term impact of selected data if the model is trained to convergence, primarily due to the prohibitive cost of full-scale LLM pretraining. In this paper, we introduce BLISS (\textbf{B}ileve\textbf{L} \textbf{I}nfluence \textbf{S}coring method for data \textbf{S}election): a lightweight data selection method that operates entirely \emph{from scratch}, without relying on any external pretrained oracle models, while explicitly accounting for the long-term impact of selected data. BLISS leverages a small proxy model as a surrogate for the LLM and employs a score model to estimate the long-term influence of training samples if the proxy model is trained to convergence. We formulate data selection as a bilevel optimization problem, where the upper-level objective optimizes the score model to assign importance weights to training samples, ensuring that minimizing the lower-level objective (i.e., training the proxy model over the weighted training loss until convergence) leads to best validation performance. Once optimized, the trained score model predicts influence scores for the dataset, enabling efficient selection of high-quality samples for LLM pretraining. We validate BLISS by pretraining 410M/1B/2.8B Pythia and LLaMA-0.5B models on selected subsets of the C4 dataset. Notably, under the 1B model setting, BLISS achieves $1.7\times$ speedup in reaching the same performance as the state-of-the-art method, demonstrating superior performance across multiple downstream tasks.

Figures (8)

• Figure 1: The pipeline of data selection and pretraining procedure. There are four main steps in one round training, 1) Warm up LLM using randomly selected training data (e.g. 10k step); 2) Bilevel optimization for score and proxy model, 3) Predict the data influence, and select Top-20% training data based on their score ranking; 4) Retrain the LLM using the selected data (e.g., 10k steps); 5) Evaluate on the downstream task. Repeating the above steps can achieve multiple-round training.
• Figure 2: The downstream performance of Pythia-410M/1B model w.r.t. pretraining FLOPs and steps, where the first point denotes the performance of a warm-up model trained on random data.
• Figure 3: The performance comparison of bilevel optimization with/without KL divergence. The number on the bar indicate the accuracy improvement compared to the method without KL divergence.
• Figure 4: The evolution of the lower-level training loss and KL divergence for different proxy model size. Subfigures (a), (b): Proxy model size 31M, target LLM size 410M. Subfigures (c), (d): Proxy model size 160M, target LLM size 410M.
• Figure 5: Comparison of BLISS trained with different validation datasets (410M model, 10B tokens). We compare our method with different validation datasets with random selection on 1 downstream task in each subplot.