Can Models Learn Skill Composition from Examples?

TL;DR

The paper investigates whether small-language models can learn compositional generalization from examples by fine-tuning on GPT-4–generated data that mix multiple language skills. Using a pipeline that generates datasets with up to k=3 skills and evaluates on held-out skills/topics up to k=5, the authors show that fine-tuning LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 yields substantial gains in composing unseen skill sets, evidencing a meta-skill rather than memorization. The improvements persist in out-of-domain evaluations and when using a different grader (Claude-3 Opus), indicating robustness beyond GPT-4 bias. Richer, higher-k training data prove more data-efficient, and the results suggest that compositional capabilities can be induced in smaller models, with implications for AI safety and beyond stochastic parrots.

Abstract

As large language models (LLMs) become increasingly advanced, their ability to exhibit compositional generalization -- the capacity to combine learned skills in novel ways not encountered during training -- has garnered significant attention. This type of generalization, particularly in scenarios beyond training data, is also of great interest in the study of AI safety and alignment. A recent study introduced the SKILL-MIX evaluation, where models are tasked with composing a short paragraph demonstrating the use of a specified $k$-tuple of language skills. While small models struggled with composing even with $k=3$, larger models like GPT-4 performed reasonably well with $k=5$ and $6$. In this paper, we employ a setup akin to SKILL-MIX to evaluate the capacity of smaller models to learn compositional generalization from examples. Utilizing a diverse set of language skills -- including rhetorical, literary, reasoning, theory of mind, and common sense -- GPT-4 was used to generate text samples that exhibit random subsets of $k$ skills. Subsequent fine-tuning of 7B and 13B parameter models on these combined skill texts, for increasing values of $k$, revealed the following findings: (1) Training on combinations of $k=2$ and $3$ skills results in noticeable improvements in the ability to compose texts with $k=4$ and $5$ skills, despite models never having seen such examples during training. (2) When skill categories are split into training and held-out groups, models significantly improve at composing texts with held-out skills during testing despite having only seen training skills during fine-tuning, illustrating the efficacy of the training approach even with previously unseen skills. This study also suggests that incorporating skill-rich (potentially synthetic) text into training can substantially enhance the compositional capabilities of models.

Figures (2)

• Figure 1: Pipeline for evaluating the generalization capability to combine skills. We split the language skill set ${\mathcal{S}}$ from yu2023skill into training skills ${\mathcal{S}}_{\text{train}}$ and held-out skills ${\mathcal{S}}_{\text{held-out}}$, and the topic set ${\mathcal{T}}$ into training topics ${\mathcal{T}}_{\text{train}}$ and held-out topics ${\mathcal{T}}_{\text{held-out}}$. The pipeline consists of three steps: (1) generate data by prompting GPT-4. The training texts contain only training skills ${\mathcal{S}}_{\text{train}}$ and training topics ${\mathcal{T}}_{\text{train}}$, and each text exhibits at most 3 skills; (2) fine-tune LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 using the generated data; (3) evaluate the fine-tuned models on held-out skills ${\mathcal{S}}_{\text{held-out}}$ and held-out topics ${\mathcal{T}}_{\text{held-out}}$ with the number of requested skills being as large as 5. See our detailed setups in \ref{['sec:pipeline']}.
• Figure 2: The success rate of different models to compose $k$ held-out skills in a short paragraph. (See the detailed definition of "Ratio of Full Marks" in \ref{['sec:eval']}.) The strongest model like GPT-4 can compose 5 skills in a short paragraph reasonably well, while smaller models struggle to compose even 3 skills. After fine-tuning, the models' ability to compose skills improves significantly.