Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ArXiv-to-Model: A Practical Study of Scientific LM Training

Anuj Gupta

TL;DR

This work presents a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics and highlights how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors.

Abstract

While frontier large language models demonstrate strong reasoning and mathematical capabilities, the practical process of training domain-specialized scientific language models from raw sources remains under-documented. In this work, we present a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics. We describe an end-to-end pipeline covering metadata filtering, archive validation, LaTeX extraction, text normalization, domain-aware tokenization, and dense transformer training under constrained compute (2xA100 GPUs). Through 24 experimental runs, we analyze training stability, scaling behavior, data yield losses, and infrastructure bottlenecks. Our findings highlight how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors. We further analyze convergence dynamics and show stable training behavior in a data-rich regime (52B pretraining tokens). Rather than proposing a novel architecture, this work provides an engineering-grounded, transparent account of training a small scientific language model from scratch. We hope these insights support researchers operating under moderate compute budgets who seek to build domain-specialized models.

ArXiv-to-Model: A Practical Study of Scientific LM Training

TL;DR

This work presents a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics and highlights how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors.

Abstract

While frontier large language models demonstrate strong reasoning and mathematical capabilities, the practical process of training domain-specialized scientific language models from raw sources remains under-documented. In this work, we present a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics. We describe an end-to-end pipeline covering metadata filtering, archive validation, LaTeX extraction, text normalization, domain-aware tokenization, and dense transformer training under constrained compute (2xA100 GPUs). Through 24 experimental runs, we analyze training stability, scaling behavior, data yield losses, and infrastructure bottlenecks. Our findings highlight how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors. We further analyze convergence dynamics and show stable training behavior in a data-rich regime (52B pretraining tokens). Rather than proposing a novel architecture, this work provides an engineering-grounded, transparent account of training a small scientific language model from scratch. We hope these insights support researchers operating under moderate compute budgets who seek to build domain-specialized models.
Paper Structure (50 sections, 2 equations, 6 figures, 3 tables)

This paper contains 50 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Training Data Scaling Considerations
  3. How much data (tokens) do you need?
  4. Converting Text Size to Tokens
  5. Practical Data Scale Regimes
  6. Metadata Filtering and Normalization
  7. Sampling Weights and Data Mixture
  8. Pretraining vs Post-Training Strategy.
  9. Tokenization
  10. Design Objectives
  11. Methodology
  12. Final Tokenizer Selection
  13. Tokenization Output Statistics
  14. Evaluation Metrics
  15. Model Architecture
  16. ...and 35 more sections

Figures (6)

  • Figure 1: Training loss for Run 24 (20GB dataset). Loss oscillates and converges slowly, indicating limited data regime.
  • Figure 2: Training loss for Run 23 (200GB dataset). Loss decreases smoothly with improved stability compared to small-data regime.
  • Figure 3: Final optimized run (Run 20, 200GB). Training loss exhibits smooth monotonic convergence with long-tail stabilization.
  • Figure 4: Training and validation loss curves for the final run. No widening divergence is observed.
  • Figure 5: Gradient norm across training steps. Stability is achieved after early warm-up.
  • ...and 1 more figures