Transferability in Deep Learning: A Survey

TL;DR

This survey addresses the data efficiency gap in deep learning by systematically analyzing transferability across the full lifecycle of model development. It organizes transfer learning into pre-training (supervised and unsupervised) and adaptation (task and domain), detailing core methods such as meta-learning, causal learning, generative and contrastive pre-training, domain adversarial training, and domain translation. A key contribution is the open-source Translearner library and a benchmark suite that enable fair, large-scale evaluation of transferability across tasks and domains. The work emphasizes practical challenges—catastrophic forgetting, negative transfer, and domain shifts—while outlining principled guidelines and theory-backed approaches to improve transferability in real-world settings.

Abstract

The success of deep learning algorithms generally depends on large-scale data, while humans appear to have inherent ability of knowledge transfer, by recognizing and applying relevant knowledge from previous learning experiences when encountering and solving unseen tasks. Such an ability to acquire and reuse knowledge is known as transferability in deep learning. It has formed the long-term quest towards making deep learning as data-efficient as human learning, and has been motivating fruitful design of more powerful deep learning algorithms. We present this survey to connect different isolated areas in deep learning with their relation to transferability, and to provide a unified and complete view to investigating transferability through the whole lifecycle of deep learning. The survey elaborates the fundamental goals and challenges in parallel with the core principles and methods, covering recent cornerstones in deep architectures, pre-training, task adaptation and domain adaptation. This highlights unanswered questions on the appropriate objectives for learning transferable knowledge and for adapting the knowledge to new tasks and domains, avoiding catastrophic forgetting and negative transfer. Finally, we implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.

Figures (24)

• Figure 1: The life cycle of most deep learning applications. The model is first pre-trained on some large-scale data and then adapted to a different target task. When labeled data is not enough, data from the same task but another distribution named source domain will also be used to boost the performance on the target task.
• Figure 2: Overview of the survey. The survey is organized around the life cycle of deep learning applications and focuses on the core problems and methods related to transferability. Besides, we briefly introduce some related learning setups.
• Figure 3: Development of pre-training methods that are related to transferablity.
• Figure 4: Illustration of manually designed inductive bias (left) and learned inductive bias from pre-training data (right). (Figure from transfer_learning.)
• Figure 5: The architecture for standard supervised pre-training. The model is composed of a feature generator and a task-specific head. The aim of supervised pre-training is to obtain a transferable feature generator from some large-scale data, which might be labeled manually in a strict pipeline, or generated in the wild with social media hashtags. After pre-training, the feature generator is adapted to the downstream tasks, while the task-specific head is usually discarded.

Theorems & Definitions (11)

• Definition 1: Transferability
• Definition 2: Negative Transfer Gap
• Theorem 3: Bound with Disparity
• Definition 4: $\mathcal{H}\Delta \mathcal{H}$-Divergence
• Theorem 5: DATheroy10
• Theorem 6: GeneralizedDATheory
• Definition 7: Disparity Discrepancy
• Theorem 8: MDD
• Definition 9: Margin Disparity Discrepancy
• Theorem 10: MDD