REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback

TL;DR

REFINE-AF introduces a three-stage, RL-based framework that leverages small open-source LLMs to generate high-quality, task-agnostic instructions and corresponding input–output triplets. By bootstrapping from a seed set, refining data via Reinforcement Learning from Automated Feedback, and producing an Instruction Fine-Tuning dataset, the approach achieves substantial improvements over prior self-instruction pipelines on the Super-NI benchmark, even with limited human labeling. The authors demonstrate strong zero-shot generalization, improved human-rated performance on user-oriented tasks, and data-size benefits, while releasing a 45K-instruction synthetic dataset to accelerate research. Overall, REFINE-AF shows that low-parameter, open-source LLMs can effectively generate diverse, high-quality instructional data at reduced cost, enabling scalable instruction-finetuning and broader task coverage.

Abstract

Instruction-based Large Language Models (LLMs) have proven effective in numerous few-shot or zero-shot Natural Language Processing (NLP) tasks. However, creating human-annotated instruction data is time-consuming, expensive, and often limited in quantity and task diversity. Previous research endeavors have attempted to address this challenge by proposing frameworks capable of generating instructions in a semi-automated and task-agnostic manner directly from the model itself. Many of these efforts have relied on large API-only parameter-based models such as GPT-3.5 (175B), which are expensive, and subject to limits on a number of queries. This paper explores the performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B, and Mistral 7B, using a semi-automated framework, thereby reducing human intervention, effort, and cost required to generate an instruction dataset for fine-tuning LLMs. Furthermore, we demonstrate that incorporating a Reinforcement Learning (RL) based training algorithm into this LLMs-based framework leads to further enhancements. Our evaluation of the dataset reveals that these RL-based frameworks achieve a substantial improvements in 63-66% of the tasks compared to previous approaches.

Figures (8)

• Figure 1: Schematic diagram of the stages in $\textsc{REFINE-AF}$ pipeline.
• Figure 2: Moving average of the model rewards over the training steps for LLaMa 13B, LLaMa 13B and Mistral 7B.
• Figure 3: Length distributions of the instructions, inputs, and outputs generated by $\textsc{REFINE-AF}$.
• Figure 4: Distribution of Rouge-L scores between generated instructions and their most similar seed instruction. This highlights the difference in generated instructions from seed tasks.
• Figure 5: Similarity of instructions generated using $\textsc{REFINE-AF}$ wrt the GPT 3.5 instructions for different base models. This similarity has been calculated using Rouge-L scores.