Instruction Tuning With Loss Over Instructions

TL;DR

This work introduces Instruction Modelling (IM), a simple extension of Instruction Tuning (IT) that applies the loss to both instruction and completion tokens, excluding template prompts. Across 21 benchmarks and multiple model scales, IM improves performance on traditional NLP tasks and open-ended generation, with notable gains when instructions are long relative to outputs and in low-resource settings consistent with the Superficial Alignment Hypothesis (SAH). Key empirical signals suggest IM mitigates overfitting to instruction tuning data, evidenced by higher train losses but lower test losses and lower BLEU-based memorization compared to IT, though KL divergence regularisation is not a reliable fix. The findings provide practical guidance for instruction tuning in low-resource scenarios and point to potential further gains when combining IM with methods like Neftune, while highlighting limitations related to data quality and ethical considerations.

Abstract

Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, we propose a simple yet effective method, Instruction Modelling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (e.g., MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (e.g., MT-Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing \ours as a replacement for current fine-tuning processes. Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios.

Figures (11)

• Figure 1: Performance differences between Instruction Tuning (IT) and our proposed method instruction modelling (IM) trained on 7 instruction tuning datasets. These datasets contain prompts and responses but do not contain preference pairs. Specifically, we use the Less datasets xia2024less and Alpagasus datasets chen2024alpagasus, which are subsets of Flan V2JMLR:v25:23-0870, DollyDollyV2, and Stanford Alpacaalpaca to ensure good performance. We also report the results on the LIMA dataset. (Left) The mean performance across 18 traditional NLP tasks (see §\ref{['para:evaluation']} for details). (Right) The win rate on the AlpacaEval 1.0 benchmark alpaca_eval. Please refer to §\ref{['sec:main']} for details.
• Figure 2: (Left) Performance improvement, achieved by our approach instruction modelling (IM) compared to Instruction Tuning (IT) on the AlpacaEval 1.0, against the ratio between average instruction length and average output length in instruction tuning datasets (training size noted in parentheses). We highlight several representative instruction tuning datasets in yellow. Our analysis suggests that IM is especially beneficial for datasets characterized by lengthy instructions or prompts paired with comparably brief outputs, such as Code Alpacacodealpaca and Less MMLU Chatxia2024less. (Right) Performance improvement achieved by our approach IM over IT on the AlpacaEval 1.0 against the number of training examples in instruction tuning datasets. Here we maintain a fixed ratio between instruction and output length of 10. This analysis suggests that IM is particularly effective under the low-resource setting or Superficial Alignment Hypothesis. Please refer to §\ref{['sec:main']} for details.
• Figure 3: (Left) Training loss distribution for each example between our approach instruction modelling (IM) and Instruction Tuning (IT) on the LIMA dataset. (Right) Test loss distribution for each example between IM and IT on the Tulu V2 dataset, using a 10% randomly sampled data for efficacy. Mean losses are marked by dashed lines. For both IM and IT, here we only compute the loss over the output part. IM has a higher train loss with lower test loss, suggesting that IM effectively mitigates the overfitting issues compared to IT. See Appendix §\ref{['sec:train_test_loss']} for more examples.
• Figure 4: Mean performance on 18 NLP tasks over epochs using Llama-2-7B-Base. This analysis suggests that IM experiences a lower instruction tuning tax compared to IT.
• Figure 5: Comparison of Instruction Tuning (IT) and instruction modelling (IM) methods using OPT-6.7B (Top Row) and Llama-2-13B-Base (Bottom Row) trained on 7 instruction tuning datasets. (Left) The mean performance across 18 NLP tasks. (Right) The win rate on the AlpacaEval 1.0 benchmark.