Technical Report on Text Dataset Distillation

TL;DR

This paper surveys text dataset distillation, a method for compressing large labeled text collections into small synthetic sets that preserve training efficacy. It outlines a four-category taxonomy (Meta-Model, Gradient, Trajectory, Distribution) and traces milestones from soft-label embeddings to discrete, model-agnostic data generation and large decoder-based distillation. It covers fairness, detoxification, and multimodal extensions, and reviews models and datasets used, highlighting a shift toward cross-architecture distillation and privacy considerations. Despite progress, the authors note gaps in benchmarks, real-world applicability, privacy guarantees, and scaling to complex tasks and large LLMs, framing a maturing but still evolving field.

Abstract

In the vision domain, dataset distillation arises as a technique to condense a large dataset into a smaller synthetic one that exhibits a similar result in the training process. While image data presents an extensive literature of distillation methods, text dataset distillation has fewer works in comparison. Text dataset distillation initially grew as an adaptation of efforts from the vision universe, as the particularities of the modality became clear obstacles, it rose into a separate branch of research. Several milestones mark the development of this area, such as the introduction of methods that use transformer models, the generation of discrete synthetic text, and the scaling to decoder-only models with over 1B parameters. Despite major advances in modern approaches, the field remains in a maturing phase, with room for improvement on benchmarking standardization, approaches to overcome the discrete nature of text, handling complex tasks, and providing explicit examples of real-world applications. In this report, we review past and recent advances in dataset distillation for text, highlighting different distillation strategies, key contributions, and general challenges.

Figures (2)

• Figure 1: Taxonomy of Dataset Distillation Methods.
• Figure 2: Timeline of milestones in text dataset distillation. From 2021 to 2025, the timeline highlights important events that shape the research area. Evolving from a simple extension of a vision work, state-of-the-art dataset distillation methods now leverage transformer models of over a billion parameters to generate cross-architecture readable distilled datasets that guarantee privacy of the original data.