ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models

TL;DR

ChemPile addresses the scarcity and fragmentation of large-scale, diverse chemical data by introducing a multimodal, open dataset designed for training and evaluating chemistry foundation models. It combines seven interrelated subsets—Education, Paper, (m)LIFT, Reasoning, Code, Caption, and Splits—across multiple representations (SMILES, SELFIES, InChI, IUPAC) and modalities (text, code, images) with expert curation and robust train/validation/test splits. The approach emphasizes scale, diversity, and quality, offering a uniform HuggingFace interface, templates, and an extensible sampling engine to enable controlled pretraining and benchmarking. By mirroring the human learning journey through foundational knowledge to advanced reasoning, ChemPile aims to catalyze chemical AI research, enable cross-domain knowledge transfer, and accelerate discovery in drug design, materials science, and climate-relevant chemistry.

Abstract

Foundation models have shown remarkable success across scientific domains, yet their impact in chemistry remains limited due to the absence of diverse, large-scale, high-quality datasets that reflect the field's multifaceted nature. We present the ChemPile, an open dataset containing over 75 billion tokens of curated chemical data, specifically built for training and evaluating general-purpose models in the chemical sciences. The dataset mirrors the human learning journey through chemistry -- from educational foundations to specialized expertise -- spanning multiple modalities and content types including structured data in diverse chemical representations (SMILES, SELFIES, IUPAC names, InChI, molecular renderings), scientific and educational text, executable code, and chemical images. ChemPile integrates foundational knowledge (textbooks, lecture notes), specialized expertise (scientific articles and language-interfaced data), visual understanding (molecular structures, diagrams), and advanced reasoning (problem-solving traces and code) -- mirroring how human chemists develop expertise through diverse learning materials and experiences. Constructed through hundreds of hours of expert curation, the ChemPile captures both foundational concepts and domain-specific complexity. We provide standardized training, validation, and test splits, enabling robust benchmarking. ChemPile is openly released via HuggingFace with a consistent API, permissive license, and detailed documentation. We hope the ChemPile will serve as a catalyst for chemical AI, enabling the development of the next generation of chemical foundation models.

Figures (7)

• Figure 1: Overview of the ChemPile and its curation process. The figure illustrates the dataset creation process. Education and Caption consist of gathering resources from online resources. Code and (m)LIFT are based on dataset content, for the first filtering from general datasets, while for the second, by filling templates with the data. For ChemPile-Paper, the content is collected by filtering and processing published open-source papers. Finally, reasoning is based on distilling knowledge from LLMs and processing data from Stack Exchange. The resulting datasets are published in a format that is very easy to use on HuggingFace.
• Figure 2: (a): Token count comparison between the ChemPile dataset and other domain-specific large datasets used to train foundation models.ChemDFMzhao2024chemdfm is a foundation model for chemistry trained on 34B tokens in chemistry-related papers and textbooks augmented with general text (49M tokens), BatGPT yang2024batgpt is a foundation model for chemical engineering, BioGPT luo2022biogpt for biology, and ChemGPT frey2023neural is a foundation model trained only on SMILES string. LlaSMol yu2024llasmol0 is an instruction-tuning dataset for chemistry. ChemDual lin2025enhancing is a 4.4 million instruction dataset for chemical reactions. The value for BioGPT is an estimate based on: 15 M abstracts × 250 words × 1.2 tokens $\approx$ 4.5 B tokens. We compute the estimate for LlaSMol based on the published HuggingFace dataset. The scale of our dataset exceeds any of the corpora used to pre-train or fine-tune existing chemistry LLMs. (b): Embedded datapoints sampled from various subsets of ChemPile vs other public datasets. Note, only the instruction tuning data made public by the authors of Darwin xie2024darwin is used. We embed only the first 512 tokens of each sampled document using the specter2-base model provided by Singh2022SciRepEvalAM. Along PCA, we provide UMAP and TSNE plots in \ref{['app:tsne_umap']}.
• Figure 3: Correlation between the Tanimoto similarity and the cosine similarity ($\theta_{sim}$) of the embeddings for most common chemical representations. The correlation is shown for four representation embeddings: SMILES (top left), IUPAC name (top right), SELFIES (bottom left), and InChI (bottom right). For the four subplots, we show the Pearson correlation r in the top left corner of all subplots.
• Figure 4: Example of how our sampling engine operates. The sampling depends on two a metadata file, and a raw data file containing all the correct columns as described in the metadata. The colors match what elements of the text templates is replaced in the final text with natural language.
• Figure 5: ChemPile-Education covers different kinds of educational data. Textbook data contains foundational knowledge, but also worked examples.