Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain-Adaptive Continued Pre-Training of Small Language Models

Salman Faroz

TL;DR

The paper addresses adapting small language models to educational content under constrained compute by performing domain-specific continued pre-training on a 125M-parameter model. It employs memory-efficient, incremental training from 400M to 1B tokens, with sequence packing and streaming data to maximize throughput. Key findings include substantial gains on knowledge and reasoning benchmarks (e.g., MMLU +8.1%, HellaSwag +7.6%) alongside domain-specialization trade-offs that degrade some general tasks, and clear evidence of token-efficiency gains with modest data budgets. The work demonstrates that targeted, resource-conscious continued pre-training can yield meaningful domain specialization for small LMs, offering a practical path for domain-adaptive NLP in settings with limited computational resources.

Abstract

Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources. I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch. Using a 125M parameter model, I demonstrate significant performance improvements through incremental training on 400 million tokens, followed by further training to reach 1 billion tokens. My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation. Results show notable gains in knowledge-intensive tasks (MMLU +8.1%) and contextual understanding (HellaSwag +7.6%), while revealing educational domain specialization trade-offs. I analyze token efficiency, catastrophic forgetting mitigation strategies, and scaling patterns. My findings suggest that thoughtful preprocessing and training methodologies enable meaningful improvements in language model capabilities even with constrained computational resources, opening pathways for domain-specific adaptation of smaller language models.

Domain-Adaptive Continued Pre-Training of Small Language Models

TL;DR

The paper addresses adapting small language models to educational content under constrained compute by performing domain-specific continued pre-training on a 125M-parameter model. It employs memory-efficient, incremental training from 400M to 1B tokens, with sequence packing and streaming data to maximize throughput. Key findings include substantial gains on knowledge and reasoning benchmarks (e.g., MMLU +8.1%, HellaSwag +7.6%) alongside domain-specialization trade-offs that degrade some general tasks, and clear evidence of token-efficiency gains with modest data budgets. The work demonstrates that targeted, resource-conscious continued pre-training can yield meaningful domain specialization for small LMs, offering a practical path for domain-adaptive NLP in settings with limited computational resources.

Abstract

Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources. I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch. Using a 125M parameter model, I demonstrate significant performance improvements through incremental training on 400 million tokens, followed by further training to reach 1 billion tokens. My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation. Results show notable gains in knowledge-intensive tasks (MMLU +8.1%) and contextual understanding (HellaSwag +7.6%), while revealing educational domain specialization trade-offs. I analyze token efficiency, catastrophic forgetting mitigation strategies, and scaling patterns. My findings suggest that thoughtful preprocessing and training methodologies enable meaningful improvements in language model capabilities even with constrained computational resources, opening pathways for domain-specific adaptation of smaller language models.

Paper Structure

This paper contains 36 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Pre-Training vs. Continued Pre-Training vs. Fine-Tuning
  4. Domain Adaptation in Language Models
  5. Catastrophic Forgetting in Continual Learning
  6. Token Efficiency in Model Training
  7. Methodology
  8. Data Acquisition and Selection
  9. Data Preprocessing Pipeline
  10. Quality Filtering
  11. Deduplication Strategies
  12. Tokenization Process
  13. Sequence Packing Approach
  14. Training Infrastructure
  15. Distributed Training Configuration
  16. ...and 21 more sections

Figures (2)

  • Figure 1: Training loss comparison between 400M and 1B token continued pre-training runs. The 1B token model (orange) consistently achieves lower loss than the 400M token model (green) across training steps, indicating more effective knowledge acquisition.
  • Figure 2: Token efficiency analysis showing performance trends across training volumes. Educational benchmarks (blue) show consistent improvement with increased token count, while general benchmarks (green) exhibit slight degradation, highlighting domain specialization trade-offs.