Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning

TL;DR

This work introduces TAPIR, a Task-Aware Curriculum Planning framework for distilling instruction-following abilities in LLMs. By combining an oracle LLM to identify hard instructions, a judge LLM to assess response quality, and a task-aware data expansion with multi-round curriculum planning, TAPIR creates balanced, progressively challenging training data. Across LLaMA2 and Qwen1.5 backbones, TAPIR achieves state-of-the-art or competitive results on AlpacaEval 2.0 and MT-Bench with substantially less training data than baselines, and ablations validate the contribution of MCP and task-aware data strategies. The approach highlights the importance of task distributions and curriculum design in instruction tuning and offers a scalable path for improving smaller models while mitigating overfitting to easier tasks.

Abstract

Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses. While several studies have explored autonomous approaches to distilling and annotating instructions from powerful proprietary LLMs, such as ChatGPT, they often neglect the impact of the distributions and characteristics of tasks, together with the varying difficulty of instructions in training sets. This oversight can lead to imbalanced knowledge capabilities and poor generalization powers of student LLMs. To address these challenges, we introduce Task-Aware Curriculum Planning for Instruction Refinement (TAPIR), a multi-round distillation framework that utilizes an oracle LLM to select instructions that are difficult for a student LLM to follow. To balance the student's capabilities, task distributions in training sets are adjusted with responses automatically refined according to their corresponding tasks. In addition, by incorporating curriculum planning, our approach systematically escalates the difficulty levels of tasks, progressively enhancing the student LLM's capabilities. We rigorously evaluate TAPIR using several widely recognized benchmarks (such as AlpacaEval 2.0, MT-Bench, etc.) and multiple student LLMs. Empirical results demonstrate that student LLMs, trained with our method and less training data, outperform larger instruction-tuned models and strong distillation baselines.

Figures (6)

• Figure 1: Comparison between different instruction-tuned LLaMA2-based models on the AlapcaEval 2.0 and MT-Bench benchmarks. Our resulting 7B models (TAPIR-7B-S/M) significantly outperform baselines, whose performance even exceeds that of 13B models.
• Figure 2: An overview of the TAPIR framework.
• Figure 3: Performance of TAPIR-7B on AlpacaEval 2.0 and MT-Bench through training rounds.
• Figure 4: Relative response quality against ChatGPT on diverse task categories of Vicuna-Instructions.
• Figure 5: The comparison of task distributions of our training datasets.