REInstruct: Building Instruction Data from Unlabeled Corpus

TL;DR

REInstruct tackles the cost and scalability of constructing instruction data for large language models by automatically harvesting high-quality content from unlabeled corpora. It combines a heuristic-based unlabeled-text selection, a reversely trained instruction generator, and a response rewriting step to produce robust <instruction, response> pairs, which are then used in a joint fine-tuning regime with both seed and synthetic data. Empirical results on AlpacaEval with Llama-7B show competitive performance against open-source non-distilled baselines and close to distilled models, with demonstrated improvements as synthetic data scales (8K/16K/32K). The work also provides insights into data diversity, grounding of rewritten responses, and the critical role of rewriting and selection signals in quality instruction data generation.

Abstract

Manually annotating instruction data for large language models is difficult, costly, and hard to scale. Meanwhile, current automatic annotation methods typically rely on distilling synthetic data from proprietary LLMs, which not only limits the upper bound of the quality of the instruction data but also raises potential copyright issues. In this paper, we propose REInstruct, a simple and scalable method to automatically build instruction data from an unlabeled corpus without heavy reliance on proprietary LLMs and human annotation. Specifically, REInstruct first selects a subset of unlabeled texts that potentially contain well-structured helpful and insightful content and then generates instructions for these texts. To generate accurate and relevant responses for effective and robust training, REInstruct further proposes a rewriting-based approach to improve the quality of the generated instruction data. By training Llama-7b on a combination of 3k seed data and 32k synthetic data from REInstruct, fine-tuned model achieves a 65.41\% win rate on AlpacaEval leaderboard against text-davinci-003, outperforming other open-source, non-distilled instruction data construction methods. The code is publicly available at \url{https://github.com/cs32963/REInstruct}.

Figures (6)

• Figure 1: Comparison of our automatic instruction data annotation method and previous methods.
• Figure 2: The overall framework of our instruction data annotation method.
• Figure 3: Prompt for collecting rewritten responses from ChatGPT.
• Figure 4: Sunburst visualization of verb-noun structure of seed instructions and generated instructions on our selected unlabeled text.
• Figure 5: Sunburst visualization of verb-noun structure of generated instructions on randomly sampled unlabeled text.