Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stronger Models are NOT Stronger Teachers for Instruction Tuning

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Radha Poovendran

TL;DR

The paper challenges the view that bigger or stronger response generators always make better teachers for instruction tuning, revealing the Larger Models' Paradox. It introduces the Compatibility-Adjusted Reward (CAR) to quantify how well a response generator's outputs align with a given base model, enabling prediction of effectiveness without fine-tuning. Through extensive experiments across multiple base models and response generators, the authors show open-source generators can outperform GPT-4 and that intra-family compatibility influences success. CAR outperforms traditional data-selection metrics, offering a cost-effective approach to generating high-quality instruction datasets and guiding future instruction-tuning strategies.

Abstract

Instruction tuning has been widely adopted to ensure large language models (LLMs) follow user instructions effectively. The resulting instruction-following capabilities of LLMs heavily rely on the instruction datasets used for tuning. Recently, synthetic instruction datasets have emerged as an economically viable solution to provide LLMs diverse and high-quality instructions. However, existing approaches typically assume that larger or stronger models are stronger teachers for instruction tuning, and hence simply adopt these models as response generators to the synthetic instructions. In this paper, we challenge this commonly-adopted assumption. Our extensive experiments across five base models and twenty response generators reveal that larger and stronger models are not necessarily stronger teachers of smaller models. We refer to this phenomenon as the Larger Models' Paradox. We observe that existing metrics cannot precisely predict the effectiveness of response generators since they ignore the compatibility between teachers and base models being fine-tuned. We thus develop a novel metric, named as Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response generators. Our experiments across five base models demonstrate that CAR outperforms almost all baselines.

Stronger Models are NOT Stronger Teachers for Instruction Tuning

TL;DR

The paper challenges the view that bigger or stronger response generators always make better teachers for instruction tuning, revealing the Larger Models' Paradox. It introduces the Compatibility-Adjusted Reward (CAR) to quantify how well a response generator's outputs align with a given base model, enabling prediction of effectiveness without fine-tuning. Through extensive experiments across multiple base models and response generators, the authors show open-source generators can outperform GPT-4 and that intra-family compatibility influences success. CAR outperforms traditional data-selection metrics, offering a cost-effective approach to generating high-quality instruction datasets and guiding future instruction-tuning strategies.

Abstract

Instruction tuning has been widely adopted to ensure large language models (LLMs) follow user instructions effectively. The resulting instruction-following capabilities of LLMs heavily rely on the instruction datasets used for tuning. Recently, synthetic instruction datasets have emerged as an economically viable solution to provide LLMs diverse and high-quality instructions. However, existing approaches typically assume that larger or stronger models are stronger teachers for instruction tuning, and hence simply adopt these models as response generators to the synthetic instructions. In this paper, we challenge this commonly-adopted assumption. Our extensive experiments across five base models and twenty response generators reveal that larger and stronger models are not necessarily stronger teachers of smaller models. We refer to this phenomenon as the Larger Models' Paradox. We observe that existing metrics cannot precisely predict the effectiveness of response generators since they ignore the compatibility between teachers and base models being fine-tuned. We thus develop a novel metric, named as Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response generators. Our experiments across five base models demonstrate that CAR outperforms almost all baselines.

Paper Structure

This paper contains 36 sections, 3 equations, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Synthetic Data Generation for Instruction Tuning.
  4. Metrics for Data Selection.
  5. Which Models are the most effective teachers for instruction tuning?
  6. Preliminaries
  7. Instruction Datasets.
  8. Supervised Fine-Tuning.
  9. Experimental Setup
  10. Instruction Sets.
  11. Response Generators.
  12. Base Models.
  13. Evaluation Benchmarks.
  14. Evaluation Metrics.
  15. Instruction-Tuning and Evaluation Setup.
  16. ...and 21 more sections

Figures (9)

  • Figure 1: This figure demonstrates the process of instruction tuning and the scope of this paper.
  • Figure 2: Average performance of five base models fine-tuned on various response generators across six model families. We use different colors to distinguish between model families, with darker bars indicating larger response generators within each family.
  • Figure 3: This figure demonstrates the impact of different sampling hyper-parameters when generating responses. We use Gemma-2-9b-it as the response generator. All models are supervised-fine-tuned on the Llama-3.1-Minitron-4B base model.
  • Figure 4: This figures demonstrates the response quality measured by three reward models.
  • Figure 5: Task categories of the Magpie-100K instruction set used in our study.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Definition 4.1: Effectiveness Measure of Response Generators