LM-mixup: Text Data Augmentation via Language Model based Mixup

TL;DR

This paper tackles the inefficiency of using abundant low-quality data for instruction tuning by introducing Instruction Distillation and the Mixture dataset. It proposes LM-Mixup, a two-stage training pipeline (cold-start pretraining followed by GRPO-based reinforcement learning) guided by multi-dimensional rewards (quality, semantic alignment, format) to distill multiple low-quality inputs into high-quality instruction-output pairs. Experiments show that distilling low-quality data can match or surpass full-dataset training and outperform data-selection baselines, achieving strong results with as little as 3% of the data. The findings highlight the value of carefully distilled low-quality data for efficient, scalable instruction-tuning and offer practical methods for leveraging noisy real-world data.

Abstract

Instruction tuning is crucial for aligning Large Language Models (LLMs), yet the quality of instruction-following data varies significantly. While high-quality data is paramount, it is often scarce; conversely, abundant low-quality data is frequently discarded, leading to substantial information loss. Existing data augmentation methods struggle to augment this low-quality data effectively, and the evaluation of such techniques remains poorly defined. To address this, we formally define the task of Instruction Distillation: distilling multiple low-quality and redundant inputs into high-quality and coherent instruction-output pairs. Specifically, we introduce a comprehensive data construction pipeline to create MIXTURE, a 144K-sample dataset pairing low-quality or semantically redundant imperfect instruction clusters with their high-quality distillations. We then introduce LM-Mixup, by first performing supervised fine-tuning on MIXTURE and then optimizing it with reinforcement learning. This process uses three complementary reward signals: quality, semantic alignment, and format compliance, via Group Relative Policy Optimization (GRPO). We demonstrate that LM-Mixup effectively augments imperfect datasets: fine-tuning LLMs on its distilled data, which accounts for only about 3% of the entire dataset, not only surpasses full-dataset training but also competes with state-of-the-art high-quality data selection methods across multiple benchmarks. Our work establishes that low-quality data is a valuable resource when properly distilled and augmented with LM-Mixup, significantly enhancing the efficiency and performance of instruction-tuned LLMs.

Figures (9)

• Figure 1: The goal of Instruction Distillation. Low-quality samples show issues such as typos, lack of depth, and unclear intent, each receiving low GPT ratings. After distillation, they are combined and refined into a single high-quality sample with clear, informative, and context-rich content. Ratings are on a 5-point scale. Additional case studies are provided in Appendix \ref{['sec:case_study']}.
• Figure 2: Overview of the Mixture construction pipeline. The process consists of two stages: (1) Raw data collection from Wikipedia, including segmentation and prompt-based generation of five task types; (2) High- and low-quality data construction via LLM-based scoring, multi-variant degradation, cross-topic fusion, and noisy injection, producing the final dataset for training.
• Figure 3: Overview of the training pipeline. The process involves cold-start pretraining on a subset of Mixture, followed by policy optimization using multi-dimensional rewards.
• Figure 4: Ablation study on reward components in LM-Mixup. The left figure evaluates the effect of removing the alignment reward, while the right figure shows the impact of removing the quality reward across different tasks.
• Figure 5: Effect of model scaling on performance.