When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

TL;DR

The paper addresses how finetuning indicators scale with LLM size, pretraining data, finetuning data, and finetuning method (FMT vs PET). It introduces a multiplicative joint scaling law and validates it through large-scale experiments on bilingual LLMs (1B–16B) across machine translation and multilingual summarization. Key findings show that increases in LLM size yield more benefit than pretraining data, PET parameter scaling provides limited gains, and the best finetuning method is task- and data-dependent, with LoRA often outperforming Prompt as data grows. These insights guide practical decisions on finetuning strategy and resource allocation for real-world multilingual LLM applications.

Abstract

While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.

Figures (14)

• Figure 1: Fitted single-variable scaling laws for finetuning data scaling over different LLM model sizes on WMT14 En-De. Solid lines denote fitted scaling curves. Filled circles and triangles denote fitting and held-out data points. $\Delta_h$: mean absolute deviation on the held-out data.
• Figure 2: Fitted multiplicative joint scaling laws for LLM model size and finetuning data size on WMT14 En-De, WMT19 En-Zh and MLSum. $\Delta_e/\Delta_h$: mean absolute deviation on the fitting/held-out data. $\alpha_m/beta$: scaling exponent for LLM model size/finetuning data size. We work on 1B to 16B LLM.
• Figure 3: Fitted multiplicative joint scaling laws for pretraining data size and finetuning data size on WMT14 En-De, WMT19 En-Zh and MLSum (LLM model size: 1B). $\alpha_p$: scaling exponent for pretraining data size.
• Figure 4: Fitted multiplicative joint scaling laws for PET parameter size and finetuning data size on WMT14 En-De, WMT19 En-Zh and MLSum (LLM model size: 1B). $\alpha_t$: scaling exponent for PET parameter size.
• Figure 5: Critical finetuning data sizes between different finetuning methods estimated by the fitted joint scaling law on WMT14 En-De, WMT19 En-Zh and MLSum. We use scipy.optimize.fsolve for the estimation. Critical point for "A vs. B": the finetuning data size (y-axis) at which A performs equal to B under the base model condition at x-axis. The value varies greatly across tasks.