Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs

Sultan Alrashed

TL;DR

This paper investigates how optimization dynamics, particularly the learning rate to batch size ratio, affect small language models during post-training. It adapts AllenAI's Tulu 3 pipeline to SmolLM2-1.7B, employing supervised finetuning, Direct Preference Optimization, reward modeling, and RL with verifiable rewards. The key finding is that higher $LR$ to $BS$ ratios boost complex reasoning tasks (e.g., ARC, GSM8K) on small models, while lower ratios favor pattern recognition tasks, with the scale-dependent pattern shifting at 1.7B parameters. The work introduces SmolTulu, achieving state-of-the-art performance among sub-2B models on instruction following and competitive math reasoning, and provides training recipes and ablations to guide efficient alignment of small models for broader accessibility.

Abstract

We present SmolTulu-1.7b-Instruct, referenced in this report as SmolTulu-DPO-1130, an instruction-tuned language model that adapts AllenAI's Tulu 3 post-training pipeline to enhance Huggingface's SmolLM2-1.7B base model. Through comprehensive empirical analysis using a 135M parameter model, we demonstrate that the relationship between learning rate and batch size significantly impacts model performance in a task-dependent manner. Our findings reveal a clear split: reasoning tasks like ARC and GSM8K benefit from higher learning rate to batch size ratios, while pattern recognition tasks such as HellaSwag and IFEval show optimal performance with lower ratios. These insights informed the development of SmolTulu, which achieves state-of-the-art performance among sub-2B parameter models on instruction following, scoring 67.7% on IFEval ($Δ$11%), and mathematical reasoning with 51.6% on GSM8K ($Δ$3.4%), with an alternate version achieving scoring 57.1% on ARC ($\Delta5.4%$). We release our model, training recipes, and ablation studies to facilitate further research in efficient model alignment, demonstrating that careful adaptation of optimization dynamics can help bridge the capability gap between small and large language models.

SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs

TL;DR

This paper investigates how optimization dynamics, particularly the learning rate to batch size ratio, affect small language models during post-training. It adapts AllenAI's Tulu 3 pipeline to SmolLM2-1.7B, employing supervised finetuning, Direct Preference Optimization, reward modeling, and RL with verifiable rewards. The key finding is that higher to ratios boost complex reasoning tasks (e.g., ARC, GSM8K) on small models, while lower ratios favor pattern recognition tasks, with the scale-dependent pattern shifting at 1.7B parameters. The work introduces SmolTulu, achieving state-of-the-art performance among sub-2B models on instruction following and competitive math reasoning, and provides training recipes and ablations to guide efficient alignment of small models for broader accessibility.

Abstract

We present SmolTulu-1.7b-Instruct, referenced in this report as SmolTulu-DPO-1130, an instruction-tuned language model that adapts AllenAI's Tulu 3 post-training pipeline to enhance Huggingface's SmolLM2-1.7B base model. Through comprehensive empirical analysis using a 135M parameter model, we demonstrate that the relationship between learning rate and batch size significantly impacts model performance in a task-dependent manner. Our findings reveal a clear split: reasoning tasks like ARC and GSM8K benefit from higher learning rate to batch size ratios, while pattern recognition tasks such as HellaSwag and IFEval show optimal performance with lower ratios. These insights informed the development of SmolTulu, which achieves state-of-the-art performance among sub-2B parameter models on instruction following, scoring 67.7% on IFEval (11%), and mathematical reasoning with 51.6% on GSM8K (3.4%), with an alternate version achieving scoring 57.1% on ARC (). We release our model, training recipes, and ablation studies to facilitate further research in efficient model alignment, demonstrating that careful adaptation of optimization dynamics can help bridge the capability gap between small and large language models.

Paper Structure

This paper contains 27 sections, 2 equations, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Post-Training Recipes and Instruction Tuning
  4. Direct Preference Optimization
  5. Verifiable Rewards and Reinforcement Learning
  6. Learning Rate and Batch Size Studies
  7. Efficient and Small Language Models
  8. Supervised Finetuning
  9. Dataset
  10. Training
  11. Results & Discussion
  12. Direct Preference Optimization
  13. Dataset
  14. Training
  15. Results & Discussion
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Contour analysis of learning rate and batch size effects on different evaluation metrics during supervised finetuning of SmolLM2-135M. The color scales represent scores for each metric, with black indicating higher performance. The patterns reveal task-dependent optimal ratios between learning rate and batch size.
  • Figure 2: Direct Preference Optimization objective function, where $\pi_{\text{ref}}$ is the reference model (our SFT model) and $\beta$ controls KL divergence from the reference model.