Deep transfer learning for image classification: a survey

TL;DR

This survey formalizes deep transfer learning for image classification, arguing that large, related source data substantially aids learning when target data are scarce. It introduces a continuous, similarity- and size-aware taxonomy to guide method selection and diagnose negative transfer, and surveys instance-based, feature-based, and parameter-based DTLIC approaches, including recent PETL methods and large-scale pretraining regimes like Big Transfer. The authors summarize commonly used datasets and benchmarks, discuss real-world applications, and provide practical best-practice guidelines and transferability measures to inform practitioners. They conclude by identifying knowledge gaps and outlining directions to develop robust, data-efficient image classification through DTLIC.

Abstract

Deep neural networks such as convolutional neural networks (CNNs) and transformers have achieved many successes in image classification in recent years. It has been consistently demonstrated that best practice for image classification is when large deep models can be trained on abundant labelled data. However there are many real world scenarios where the requirement for large amounts of training data to get the best performance cannot be met. In these scenarios transfer learning can help improve performance. To date there have been no surveys that comprehensively review deep transfer learning as it relates to image classification overall. However, several recent general surveys of deep transfer learning and ones that relate to particular specialised target image classification tasks have been published. We believe it is important for the future progress in the field that all current knowledge is collated and the overarching patterns analysed and discussed. In this survey we formally define deep transfer learning and the problem it attempts to solve in relation to image classification. We survey the current state of the field and identify where recent progress has been made. We show where the gaps in current knowledge are and make suggestions for how to progress the field to fill in these knowledge gaps. We present a new taxonomy of the applications of transfer learning for image classification. This taxonomy makes it easier to see overarching patterns of where transfer learning has been effective and, where it has failed to fulfill its potential. This also allows us to suggest where the problems lie and how it could be used more effectively. We show that under this new taxonomy, many of the applications where transfer learning has been shown to be ineffective or even hinder performance are to be expected when taking into account the source and target datasets and the techniques used.

Figures (5)

• Figure 1: Increase in performance on ImageNet 1K due to model size, measured by number of parameters in millions. In general, larger models result in better performance. However, there are outliers that show a decrease in performance for the largest models. This is because the largest models overfit ImageNet1k.
• Figure 2: Deep transfer learning
• Figure 3: General transfer learning taxonomy zhuang2020comprehensive
• Figure 4: Deep learning for image classification taxonomy. Shaded cells are additions to existing transfer learning and deep transfer learning taxonomies.
• Figure 5: Transfer learning taxonomy. Blue is a measure of source and target dataset similarity, where the most similar would be transferring from one set of ImageNet classes to another such as in yosinski2014transferableplested2019analysis or self-supervised pretraining on the target dataset (self-training). Red measures the target dataset size, with small being around 10,000 labeled examples and large being a million plus labeled examples. The lines between the quadrants are gray and dotted and the colors gradually merge to indicate that there is not a clear cutoff point from one quadrant to the next, rather as the target dataset increases in size between the range of around 50,000 to 200,000 more strategies for large target datasets should be employed. The number of trainable parameters in the model architecture should be considered when deciding whether to use strategies for small or large datasets. Larger models generally require more training data to avoid overfitting the target task, as highlighted in Figure \ref{['fig:Increase-in-performance']} .

Theorems & Definitions (2)

• Definition 1
• Definition 2