Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Slamming: Training a Speech Language Model on One GPU in a Day

Gallil Maimon, Avishai Elmakies, Yossi Adi

TL;DR

Slamming demonstrates that a high‑quality Generative Speech Language Model can be trained on a single $A5000$ GPU within $24$ hours by carefully optimizing initialization, architecture, data, and training objectives. The approach leverages synthetic data, text interleaving, and direct preference optimization (DPO) to boost semantic and generative capabilities beyond what compute budgets would suggest by scaling laws. The final recipe, validated on multiple model sizes and then scaled to two $A100$ GPUs for longer runs, achieves competitive performance with substantially greater compute budgets, and is released open‑source. The work provides practical guidelines and tools to broaden access to SLM research while inviting refinement of scaling laws for speech models.

Abstract

We introduce Slam, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components. We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost. We hope these insights will make SLM training and research more accessible. In the context of SLM scaling laws, our results far outperform predicted compute optimal performance, giving an optimistic view to SLM feasibility. See code, data, models, samples at - https://pages.cs.huji.ac.il/adiyoss-lab/slamming .

Slamming: Training a Speech Language Model on One GPU in a Day

TL;DR

Slamming demonstrates that a high‑quality Generative Speech Language Model can be trained on a single GPU within hours by carefully optimizing initialization, architecture, data, and training objectives. The approach leverages synthetic data, text interleaving, and direct preference optimization (DPO) to boost semantic and generative capabilities beyond what compute budgets would suggest by scaling laws. The final recipe, validated on multiple model sizes and then scaled to two GPUs for longer runs, achieves competitive performance with substantially greater compute budgets, and is released open‑source. The work provides practical guidelines and tools to broaden access to SLM research while inviting refinement of scaling laws for speech models.

Abstract

We introduce Slam, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components. We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost. We hope these insights will make SLM training and research more accessible. In the context of SLM scaling laws, our results far outperform predicted compute optimal performance, giving an optimistic view to SLM feasibility. See code, data, models, samples at - https://pages.cs.huji.ac.il/adiyoss-lab/slamming .

Paper Structure

This paper contains 22 sections, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Setup
  4. Investigations
  5. Model & Optimisation
  6. Data
  7. Text Interleaving
  8. Synthetic Data Preference Optimisation
  9. Final Recipe
  10. Increasing Compute
  11. Limitations
  12. Conclusion
  13. Acknowledgements.
  14. Full Recipe
  15. Model Sizes
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Comparing Topic-StoryCloze performance of different as a function of training compute. Model size is indicated by the size of the circle.
  • Figure 2: Comparing validation PPL of different models of similar parameter count, with and without TWIST initialisation.
  • Figure 3: Comparing PPL of different models under TWIST initialisation.
  • Figure 4: Comparing validation PPL of our best model with different optimisers and schedulers.
  • Figure 5: Analysing the optimal part of the 24 hour compute budget that should be used for DPO, with the rest used for pre-training.