Investigating the Impact of Data Selection Strategies on Language Model Performance

TL;DR

Data selection for pretraining language models is studied by comparing how subsets drawn from a raw distribution align with a target distribution. The authors propose three strategies: random selection, DSIR using hashed n-gram features, and Hybrid Importance Resampling (HIR) that fuses hashed n-gram and neural embedding features to guide resampling. Across experiments on the Pile and six GLUE tasks, DSIR generally yields strong token-level alignment, while HIR provides complementary improvements by leveraging semantic information from neural features. The work underscores the value of data-centric choices for pretraining efficiency and generalization and suggests future work on balancing feature contributions and scaling up.

Abstract

Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{https://github.com/jgu13/HIR-Hybrid-Importance-Resampling-for-Language-Models}{github repository}.

Figures (1)

• Figure 1: Data selection and pretraining pipeline. The process begins with target and raw datasets, where a subset of raw data is selected to match the target distribution. The selected data is used for pretraining, followed by fine-tuning and evaluation on GLUE tasks