Where to start? Analyzing the potential value of intermediate models

TL;DR

The paper investigates intertraining, the practice of starting target-task finetuning from a finetuned intermediate, across diverse English classification tasks. It formalizes gains with $gain(m,t)=s_m^t-s_{PT}^t$ and analyzes how target sensitivity and base-model quality contribute to intertraining success, finding a surprising decoupling between source and target effects. A simple linear probing strategy on MNLI serves as a robust static predictor of intertraining gains, enabling a practical ranking of intermediate starters without exhaustive search. The authors propose actionable recommendations and publish updating rankings by architecture to guide real-world intertraining, highlighting that alignment between source and target is not the sole determinant and that architecture and data sizing dynamics substantially shape outcomes.

Abstract

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. This is in contrast to current perception that the alignment between the target dataset and the source dataset used to generate the base model is a major factor in determining intertraining success. We analyze different aspects that contribute to each. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture https://ibm.github.io/model-recycling/.

Figures (13)

• Figure 1: Results of in-house models/targets experiment. Columns correspond to target datasets and Rows correspond to intermediate models generated based on same datasets as source. The $22$ datasets come from the General, NLI and Twitter groups. Each value indicates intertraining gain w.r.t. the PT model, averaged over 5 seeds. Sorted by group and source average gain (bottom row). Positive significant cells (>2 STD) are italicized.
• Figure 2: Linear probing MNLI (x) is enough to predict finetuning gains (y) averaged over 14 General datasets. Each point corresponds to one off-the-shelf base model.
• Figure 3: For 'good' sources the average gain increase as the source training size increases, while for 'bad' sources it decreases.
• Figure 4: The average gain across targets decreases as the target training size increases.
• Figure 5: Standard deviation of in-house models/targets experiment. Rows correspond to intermediate models, generated based on 22 source datasets from the General, NLI and Twitter groups. Columns correspond to the same datasets, now being used as target datasets. Each value indicates standard deviation over 5 seeds.