Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

Ajay Patel, Colin Raffel, Chris Callison-Burch

TL;DR

DataDreamer addresses reproducibility bottlenecks in LLM research by delivering an open source Python library that unifies synthetic data generation, evaluation, tuning, and alignment workflows. It provides a standardized API, automatic caching and reproducibility fingerprints, and facilities for publishing open data and models to community hubs. The paper details support for multi-stage pipelines, multi-GPU training, adapters, and environment-agnostic code to reduce reimplementation effort. The work argues that such tooling can accelerate open science and enable more reliable, reusable LLM research even in the presence of closed-source models.

Abstract

Large language models (LLMs) have become a dominant and important tool for NLP researchers in a wide range of tasks. Today, many researchers use LLMs in synthetic data generation, task evaluation, fine-tuning, distillation, and other model-in-the-loop research workflows. However, challenges arise when using these models that stem from their scale, their closed source nature, and the lack of standardized tooling for these new and emerging workflows. The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them. In this paper, we introduce DataDreamer, an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science and reproducibility. The library and documentation are available at https://github.com/datadreamer-dev/DataDreamer .

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

TL;DR

DataDreamer addresses reproducibility bottlenecks in LLM research by delivering an open source Python library that unifies synthetic data generation, evaluation, tuning, and alignment workflows. It provides a standardized API, automatic caching and reproducibility fingerprints, and facilities for publishing open data and models to community hubs. The paper details support for multi-stage pipelines, multi-GPU training, adapters, and environment-agnostic code to reduce reimplementation effort. The work argues that such tooling can accelerate open science and enable more reliable, reusable LLM research even in the presence of closed-source models.

Abstract

Large language models (LLMs) have become a dominant and important tool for NLP researchers in a wide range of tasks. Today, many researchers use LLMs in synthetic data generation, task evaluation, fine-tuning, distillation, and other model-in-the-loop research workflows. However, challenges arise when using these models that stem from their scale, their closed source nature, and the lack of standardized tooling for these new and emerging workflows. The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them. In this paper, we introduce DataDreamer, an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science and reproducibility. The library and documentation are available at https://github.com/datadreamer-dev/DataDreamer .
Paper Structure (34 sections, 2 figures, 3 tables)

This paper contains 34 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. LLM Workflows
  3. Synthetic Data Generation
  4. LLMs for Task Evaluation
  5. Fine-tuning and Alignment
  6. Self-improving LLMs
  7. Demonstration and Examples
  8. DataDreamer
  9. Installation
  10. Sessions
  11. Steps
  12. Models
  13. Trainers
  14. Caching and Sharing Workflows
  15. Resumability
  16. ...and 19 more sections

Figures (2)

  • Figure 1: DataDreamer helps researchers implement many types of LLM workflows easier and makes reproducibility automatic and simple. These workflows often involve synthetic data generation with a LLM-in-the-loop and/or fine-tuning, aligning, and distilling models.
  • Figure 2: DataDreamer logs produced by the workflow in Example \ref{['example:abstract_to_tweet']} when resuming from a prior interrupted run.