Deep Transfer Learning: Model Framework and Error Analysis
Yuling Jiao, Huazhen Lin, Yuchen Luo, Jerry Zhijian Yang
TL;DR
This work develops a deep transfer-learning framework that leverages multi-domain upstream data to improve a downstream task with limited samples by learning a low-dimensional, domain-invariant representation $\mathbf{h}^*(\mathbf{X})$ and a downstream correction $Q^*(\mathbf{X})$. It jointly optimizes upstream representation learning with sparsity across domains and a downstream fine-tuning stage that enforces independence between upstream features and downstream information, enabling no, partial, or complete transfer. Theoretical results establish excess risk bounds for upstream training and downstream prediction that adapt to transfer difficulty via smoothness $\beta$, dimension reduction $d^*$, and network capacity, with concrete rates such as $\tilde{O}(n^{-\frac{\beta}{2(d+1+\beta)}})$ (upstream) and $\tilde{O}(n^{-\frac{\beta}{2(d+1+\beta)}}) + O(m^{-\frac{\beta}{2(d^*+1+2\beta)}})$ (downstream). Experiments on four image-domain datasets and one regression task validate the method, showing improvements over ERM and several baselines, and illustrating the impact of the independence penalty and representation-dimension choices. Overall, the paper advances a principled approach to deep, multi-domain transfer with interpretable feature transfer and adaptive convergence behavior.
Abstract
This paper presents a framework for deep transfer learning, which aims to leverage information from multi-domain upstream data with a large number of samples $n$ to a single-domain downstream task with a considerably smaller number of samples $m$, where $m \ll n$, in order to enhance performance on downstream task. Our framework offers several intriguing features. First, it allows the existence of both shared and domain-specific features across multi-domain data and provides a framework for automatic identification, achieving precise transfer and utilization of information. Second, the framework explicitly identifies upstream features that contribute to downstream tasks, establishing clear relationships between upstream domains and downstream tasks, thereby enhancing interpretability. Error analysis shows that our framework can significantly improve the convergence rate for learning Lipschitz functions in downstream supervised tasks, reducing it from $\tilde{O}(m^{-\frac{1}{2(d+2)}}+n^{-\frac{1}{2(d+2)}})$ ("no transfer") to $\tilde{O}(m^{-\frac{1}{2(d^*+3)}} + n^{-\frac{1}{2(d+2)}})$ ("partial transfer"), and even to $\tilde{O}(m^{-1/2}+n^{-\frac{1}{2(d+2)}})$ ("complete transfer"), where $d^* \ll d$ and $d$ is the dimension of the observed data. Our theoretical findings are supported by empirical experiments on image classification and regression datasets.