Dynamic Skill Adaptation for Large Language Models

TL;DR

The paper tackles the challenge that large language models struggle to acquire specialized, complex skills from general pretraining data. It introduces Dynamic Skill Adaptation (DSA), which constructs a skill graph by decomposing complex skills into prerequisites and then automatically generates both textbook-like descriptions for pre-training and exercise problems for instruction tuning, arranged according to human learning pathways. A key innovation is the dynamic training loop, which monitors learning with per-example metrics $\hat{L}_i$ and $\hat{\sigma}_i$, categorizes data into Data with errors, Hard-to-learn, Easy-to-learn, and Ambiguous using baselines $L_b$ and $\sigma_b$ along with $L_{average}$ and $\sigma_{average}$, and updates the training set to emphasize harder or more diverse content. Empirical results on LLAMA2 and Mistral show substantial improvements in calculus and social studies, including a ~304% improvement over general baselines and ~10.7% over specialized baselines, with strong generalization to other math benchmarks and arithmetic tasks. Overall, DSA demonstrates that coupling skill-aware data generation with dynamic curriculum-like updates can significantly enhance domain adaptation for LLMs and afford scalable learning of novel skills across domains.

Abstract

We present Dynamic Skill Adaptation (DSA), an adaptive and dynamic framework to adapt novel and complex skills to Large Language Models (LLMs). Compared with previous work which learns from human-curated and static data in random orders, we propose to first automatically generate and organize the training data by mimicking the learning pathways of human and then dynamically tailor the training data based on the training dynamics. Specifically, inspired by the learning structures and teaching strategies in the human education system, we first construct a skill graph by decomposing complex skills into sub-skills and arranging them based on their dependencies in human syllables. For every skill, we utilize LLMs to generate both textbook-like data which contains detailed descriptions of skills for pre-training and exercise-like data which targets at explicitly utilizing the skills to solve problems for instruction-tuning. Furthermore, during the instruction-tuning, we dynamically update the training data which down-weight easy-to-learn examples, generate more complex examples, and filter out data with errors. Experiments on large language models such as LLAMA and Mistral demonstrate the effectiveness of our proposed methods in adapting math reasoning skills and social study skills.

Figures (4)

• Figure 1: The overall process of our Dynamic Skill Adaptation framework. For a given complex skills, we first built the skill graph where sub-skills are organized following their dependencies (e.g., mastering summing first and then learning multiplying). Then we generate textbook-like descriptions for every skill and generate exercise-data where the skills that have been learned need to be explicitly used to solve the generated problems. During the training, we dynamically adjust the training data based on the training dynamic.
• Figure 2: The accuracy on Pre-Calculus evaluation set of every intermediate step when learning textbooks with LLAMA2-7b. The blue line (left to right) represents the process where we arrange the learning sequence following the constructed skill graph from lower-levels to higher-levels while the orange line (right to left) represents the process where the model is learning the textbook in a reversed order.
• Figure 3: A sub-skill graph in our constructed Calculus skill graph.
• Figure 4: A sub-skill graph in our constructed Social Study skill graph.