Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison

TL;DR

This work introduces open-sci-ref, a family of dense transformer baselines up to 1.7B parameters trained on up to 1T tokens across eight open reference datasets, with intermediate checkpoints to study training dynamics. By standardizing datasets, tokenization, architectures, and hyperparameters, it enables reproducible cross-scale and cross-dataset comparisons and validates procedures against HuggingFace baselines. The results reveal robust dataset rankings across scales, highlight Nemotron-CC-HQ and DCLM as top references in many settings, and provide nuanced insights from multilingual benchmarks, including the impact of language mix on English benchmarks. The release includes logs, code, and downstream evaluations to facilitate reproduction and future scaling-law research, while outlining directions for expanding licensing, mixture-of-experts architectures, and rigorous scaling analyses.

Abstract

We introduce open-sci-ref, a family of dense transformer models trained as research baselines across multiple model (0.13B to 1.7B parameters) and token scales (up to 1T) on 8 recent open reference datasets. Evaluating the models on various standardized benchmarks, our training runs set establishes reference points that enable researchers to assess the sanity and quality of alternative training approaches across scales and datasets. Intermediate checkpoints allow comparison and studying of the training dynamics. The established reference baselines allow training procedures to be compared through their scaling trends, aligning them on a common compute axis. Comparison of open reference datasets reveals that training on NemoTron-CC HQ consistently outperforms other reference datasets, followed by DCLM-baseline and FineWeb-Edu. In addition to intermediate training checkpoints, the release includes logs, code, and downstream evaluations to simplify reproduction, standardize comparison, and facilitate future research.

Figures (7)

• Figure 1: Comparison of average performance across 11 evaluation benchmarks for open-sci-ref models trained on 8 datasets with 300B tokens at different model scales. Dataset rankings remain consistent across scales, with differences becoming more pronounced at larger scales.
• Figure 2: Downstream performance across tasks of open-sci-ref models with 1.7B parameters trained on 300B tokens over different datasets. While some tasks yield clear dataset rankings (e.g., ARC, Hellaswag, Lambada), others provide insufficient signal for meaningful dataset comparison.
• Figure 3: Downstream performance across tasks of open-sci-ref 1.7B models trained for 1T tokens on the top 3 reference datasets. Nemotron ranks highest across tasks, except on Lambada.
• Figure 4: Scaling trends of open-sci-ref-0.01 baselines: 0.13B (300BT), 0.4B (300BT), 1.3B (300BT), 1.7B (300BT), and 1.7B (1TT) trained across 8 datasets (solid lines). Dashed lines show comparison with external models across parameter and token scales.
• Figure 5: Performance of English pre-trained open-sci-ref on MMMLU, which is a version of the MMLU benchmark translated to multiple languages by OpenAI employing human professional translators.