Why pre-training is beneficial for downstream classification tasks?

TL;DR

The paper addresses why pre-training improves downstream classification by introducing a game-theoretic, interaction-based representation of DNN knowledge, where $v(\boldsymbol{x})$ is the network output and $I(S|\boldsymbol{x})$ are salient interactions contributing to decisions. It quantifies how pre-trained knowledge is preserved, discarded, or newly learned during fine-tuning using a linear probe to isolate downstream-relevant knowledge $v_{pretrain}$, and decomposes interactions into $K_{preserve}$, $K_{discard}$, and $K_{new}$. The key findings show that only a small, discriminative subset of pre-trained knowledge is retained, most knowledge is discarded, and pre-training accelerates convergence by guiding the model to learn target knowledge more directly, as evidenced by preserved-learnability ratios (13%–45%) and faster, more stable increases in interaction similarity during training. These insights provide a principled understanding of pre-training and offer guidance for designing pre-training and fine-tuning pipelines to maximize downstream performance.

Abstract

Pre-training has exhibited notable benefits to downstream tasks by boosting accuracy and speeding up convergence, but the exact reasons for these benefits still remain unclear. To this end, we propose to quantitatively and explicitly explain effects of pre-training on the downstream task from a novel game-theoretic view, which also sheds new light into the learning behavior of deep neural networks (DNNs). Specifically, we extract and quantify the knowledge encoded by the pre-trained model, and further track the changes of such knowledge during the fine-tuning process. Interestingly, we discover that only a small amount of pre-trained model's knowledge is preserved for the inference of downstream tasks. However, such preserved knowledge is very challenging for a model training from scratch to learn. Thus, with the help of this exclusively learned and useful knowledge, the model fine-tuned from pre-training usually achieves better performance than the model training from scratch. Besides, we discover that pre-training can guide the fine-tuned model to learn target knowledge for the downstream task more directly and quickly, which accounts for the faster convergence of the fine-tuned model.

Figures (4)

• Figure 1: (a) We use the interaction between different input variables to represent knowledge encoded by a DNN, because the network output is proven to be well explained as the sum of numerical contributions $I(S|\boldsymbol{x})$ of interactions. (b) Explaining benefits of pre-training by analyzing effects of pre-trained model's knowledge on the downstream task.
• Figure 2: The preserved knowledge (interaction) $K^{(i)}_{\text{preserve}}$, the discarded knowledge $K^{(i)}_{\text{discard}}$, and the newly-learned knowledge $K^{(i)}_{\text{new}}$. For each subfigure, the total length of the blue bar and the orange bar equals to the knowledge encoded by the pre-trained model $K^{(i)}_{\text{pretrain}}$, and the length of the green bar and the orange bar equals to the knowledge encoded by the fine-tuned model $K^{(i)}_{\text{finetune}}$.
• Figure 3: The ratio of the preserved knowledge that can be learned by the model training from scratch. This figure verifies that pre-training makes the fine-tuned model encodes more exclusively-learned and discriminative knowledge for inference than the model training from scratch, which responses to the superior performance of the fine-tuned model.
• Figure 4: Changes of the Jaccard similarity $\text{Jaccard}_{\text{finetune}}$ and $\text{Jaccard}_{\text{finetune}}$ along with the epoch number. The similarity $\text{Jaccard}_{\text{finetune}}$ of the fine-tuned model exhibits a more sharp and stable increase with the epoch number than that of training from scratch $\text{Jaccard}_{\text{finetune}}$. This verifies the fine-tuned model learns target knowledge more quickly and directly, which accounts for its faster convergence.

Theorems & Definitions (1)

• Theorem 3.1: universal-matching property of interactions