Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data

TL;DR

TESR tackles transfer learning under limited target data and domain heterogeneity by learning a sufficient and invariant representation from multiple sources and then augmenting it with a target-specific component. The framework decouples source knowledge from target adaptation via independence constraints, enabling cross-domain transfer even when source and target tasks differ in form. Theoretical excess-risk guarantees and extensive simulations and real-data experiments demonstrate that TESR often outperforms traditional transfer methods, with practical gains in gene expression and image classification tasks. This approach offers a flexible, representation-based paradigm for robust knowledge transfer across diverse supervised learning problems.

Abstract

Transfer learning is an important approach for addressing the challenges posed by limited data availability in various applications. It accomplishes this by transferring knowledge from well-established source domains to a less familiar target domain. However, traditional transfer learning methods often face difficulties due to rigid model assumptions and the need for a high degree of similarity between source and target domain models. In this paper, we introduce a novel method for transfer learning called Transfer learning through Enhanced Sufficient Representation (TESR). Our approach begins by estimating a sufficient and invariant representation from the source domains. This representation is then enhanced with an independent component derived from the target data, ensuring that it is sufficient for the target domain and adaptable to its specific characteristics. A notable advantage of TESR is that it does not rely on assuming similar model structures across different tasks. For example, the source domain models can be regression models, while the target domain task can be classification. This flexibility makes TESR applicable to a wide range of supervised learning problems. We explore the theoretical properties of TESR and validate its performance through simulation studies and real-world data applications, demonstrating its effectiveness in finite sample settings.

Figures (13)

• Figure 1: The illustration of the proposed framework. The sufficient and invariant representation $R_c$ summarizes information from the sources, while $R_t$ captures specific information relevant to the target task, ensuring that $[R_c, R_t]$ is sufficient for the target data. Importantly, $R_t(X_0)$ and $R_c(X_0)$ are required to be independent.
• Figure 2: The relationships between $R_c$, $R_t$, and $R_0$. Estimating $R_t$ is simpler than estimating $R_0$ when $R_c \not\perp\!\!\!\perp R_0$ (left panel) with dataset $\mathcal{D}_0$ because $R_c$ contains useful information about $R_0$. Conversely, when source data does not contain useful information about the target data, meaning $R_c \;\, \space \space \;\, R_0$ (right panel), estimating $R_t$ is similar to directly estimating $R_0$.
• Figure 3: Classification accuracy and its standard deviation (represented by the width of the error bar) for the four methods, TESR, DNN, DDR, and TransIRM, are evaluated with various values of $(n_s,n_0,d)$ over 100 replications in Example 1.
• Figure 4: Classification accuracy and its standard deviation (width of error bar) on the two independent target tasks in Example 2.
• Figure 5: The box plot illustrates classification accuracy over 100 replications on the target dataset in Example 3, under $L_1$ (left panel) and cosine distance departure (right panel) conditions, where source datasets are sequentially added into the modeling process. TESR (notched box with solid line) outperforms TransIRM (notched box with solid line), DDR (rectangular box with solid line), and DNN (rectangular box with dashed line). DDR and DNN are only presented once, as they do not utilize the source information.

Theorems & Definitions (7)

• Lemma 4.1: Convergence result of learning representation on sources
• Remark 1
• Theorem 1: Convergence result of TESR on the target domain
• Definition 1
• Definition 2: Anisotropic Besov space $B^{\boldsymbol{\beta}}_{p,q,\tilde{\beta}}(\Omega)$ suzuki2021deep
• Lemma B.1
• Proof 1