Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

stable-pretraining-v1: Foundation Model Research Made Simple

Randall Balestriero, Hugues Van Assel, Sami BuGhanem, Lucas Maes

TL;DR

This work addresses the heavy engineering burden that constrains foundation-model and self-supervised learning (SSL) research, such as large-scale data handling, complex training loops, and disparate evaluation pipelines. It introduces stable-pretraining, a modular library built atop PyTorch, Lightning, Hugging Face, and TorchMetrics that unifies SSL utilities, including probes, collapse detection metrics, augmentation pipelines, and evaluation routines, under a log-everything framework. The paper details core design choices—a Manager coordinating training, a dictionary-first data/module interface, and plug-and-play callbacks for monitoring—alongside demonstrations such as depth-wise representation probing and analysis of CLIP degradation under synthetic-data fine-tuning. Collectively, stable-pretraining aims to accelerate discovery, improve reproducibility, and lower the barrier to entry for large-scale foundation-model research, enabling rapid iteration and broader exploration of new ideas.

Abstract

Foundation models and self-supervised learning (SSL) have become central to modern AI, yet research in this area remains hindered by complex codebases, redundant re-implementations, and the heavy engineering burden of scaling experiments. We present stable-pretraining, a modular, extensible, and performance-optimized library built on top of PyTorch, Lightning, Hugging Face, and TorchMetrics. Unlike prior toolkits focused narrowly on reproducing state-of-the-art results, stable-pretraining is designed for flexibility and iteration speed: it unifies essential SSL utilities--including probes, collapse detection metrics, augmentation pipelines, and extensible evaluation routines--within a coherent and reliable framework. A central design principle is logging everything, enabling fine-grained visibility into training dynamics that makes debugging, monitoring, and reproducibility seamless. We validate the library by demonstrating its ability to generate new research insights with minimal overhead, including depthwise representation probing and the analysis of CLIP degradation under synthetic data finetuning. By lowering barriers to entry while remaining scalable to large experiments, stable-pretraining aims to accelerate discovery and expand the possibilities of foundation model research.

stable-pretraining-v1: Foundation Model Research Made Simple

TL;DR

This work addresses the heavy engineering burden that constrains foundation-model and self-supervised learning (SSL) research, such as large-scale data handling, complex training loops, and disparate evaluation pipelines. It introduces stable-pretraining, a modular library built atop PyTorch, Lightning, Hugging Face, and TorchMetrics that unifies SSL utilities, including probes, collapse detection metrics, augmentation pipelines, and evaluation routines, under a log-everything framework. The paper details core design choices—a Manager coordinating training, a dictionary-first data/module interface, and plug-and-play callbacks for monitoring—alongside demonstrations such as depth-wise representation probing and analysis of CLIP degradation under synthetic-data fine-tuning. Collectively, stable-pretraining aims to accelerate discovery, improve reproducibility, and lower the barrier to entry for large-scale foundation-model research, enabling rapid iteration and broader exploration of new ideas.

Abstract

Foundation models and self-supervised learning (SSL) have become central to modern AI, yet research in this area remains hindered by complex codebases, redundant re-implementations, and the heavy engineering burden of scaling experiments. We present stable-pretraining, a modular, extensible, and performance-optimized library built on top of PyTorch, Lightning, Hugging Face, and TorchMetrics. Unlike prior toolkits focused narrowly on reproducing state-of-the-art results, stable-pretraining is designed for flexibility and iteration speed: it unifies essential SSL utilities--including probes, collapse detection metrics, augmentation pipelines, and extensible evaluation routines--within a coherent and reliable framework. A central design principle is logging everything, enabling fine-grained visibility into training dynamics that makes debugging, monitoring, and reproducibility seamless. We validate the library by demonstrating its ability to generate new research insights with minimal overhead, including depthwise representation probing and the analysis of CLIP degradation under synthetic data finetuning. By lowering barriers to entry while remaining scalable to large experiments, stable-pretraining aims to accelerate discovery and expand the possibilities of foundation model research.

Paper Structure

This paper contains 12 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. stable-pretraining: An Overview
  3. Structure
  4. Manager and logging-everything.
  5. Dictionary-first design.
  6. Data and module composition.
  7. Callbacks.
  8. Accelerating Research
  9. Depth-wise representation probing.
  10. CLIP degradation under synthetic data fine-tuning.
  11. Conclusion
  12. Code snippets.

Figures (2)

  • Figure 1: Overview of stable-pretraining.
  • Figure 2: Depth-wise representation probing (ImageNet-100). We report the top-1 and top-5 validation accuracies of linear probes from different layers of SOTA vision self-supervised learning methods after 100 epochs. MetaCLIP outperforms other approaches on beginning and intermediate layers, while DINOv2-3 outperforms on the last layer.