RAISE: Reinforced Adaptive Instruction Selection For Large Language Models

TL;DR

RAISE tackles the instruction selection problem in LLM fine-tuning by replacing static heuristic filtering with a dynamic, task-objective-driven acquisition function learned through reinforcement learning. It models instruction selection as a sequential decision process and uses state fusion to incorporate training progress and data features, with diversity enforced via clustering. Using PPO with GAE, it optimizes the acquisition function to maximize final performance, achieving notable data efficiency (only $1\%$ of gradient-update steps) while outperforming static baselines across multiple models and tasks, including GSM8K. This approach enables targeted, data-efficient alignment of LLMs to specific objectives, offering scalable benefits for instruction tuning and potential generalization to diverse task regimes.

Abstract

In the instruction fine-tuning of large language models (LLMs), it is widely recognized that a few high-quality instructions are superior to a large number of low-quality instructions. At present, many instruction selection methods have been proposed, but most of these methods select instruction based on heuristic quality metrics, and only consider data selection before training. These designs lead to insufficient optimization of instruction fine-tuning, and fixed heuristic indicators are often difficult to optimize for specific tasks. Therefore, we design a dynamic, task-objective-driven instruction selection framework RAISE(Reinforced Adaptive Instruction SElection), which incorporates the entire instruction fine-tuning process into optimization, selecting instructions at each step based on the expected impact of each instruction on model performance improvement. Our approach is well interpretable and has strong task-specific optimization capabilities. By modeling dynamic instruction selection as a sequential decision-making process, we use RL to train our selection strategy. Extensive experiments and result analysis prove the superiority of our method compared with other instruction selection methods. Notably, RAISE achieves superior performance by updating only 1% of the training steps compared to full-data training, demonstrating its efficiency and effectiveness.

Figures (8)

• Figure 1: Overview of the RAISE framework, illustrating the training process of LLM at step $t$: (1) The source instruction dataset and current training state are fused to construct $s_t$, which encodes both data features and training progress. (2) The Selector, guided by the acquisition function (a trainable scorer), takes $s_t$ as input and selects a batch of instruction data. (3) This selected batch is used to update LLM, resulting in performance improvement $r_t$. (4) Finally, the updated training state forms $s_{t+1}$, serving as input for the next step.
• Figure 2: Performance of GSM8K-targeted training. All methods are still trained on Alpaca dataset but use GSM8K as validation set (different from the GSM8K evaluation set). Besides, We use MathQA and MMLU.Math (MMLU.college_mathematics) for OOD evaluations.
• Figure 3: Performance with different instruction semantic dimensions $d_{sem}$.
• Figure 4: Ablation results on different components in state fusion.
• Figure 5: Performance with different class counts $C$ when using only 1% and 5% data. The larger the $C$, the finer the class, and the fewer instructions each class selects.