Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Residual Feature Integration is Sufficient to Prevent Negative Transfer

Yichen Xu, Ryumei Nakada, Linjun Zhang, Lexin Li

TL;DR

Negative transfer from mismatched source representations limits transfer learning. ReFine remedies this by freezing $f_ ext{rep}$, learning a target residual encoder $h$, and training a shallow adapter on the joint features $(f_ ext{rep}(x), h(x))$, ensuring performance is never worse than target-only training. The authors provide a risk bound showing a favorable trade-off between a parametric term and a nonparametric term that improves as the residual becomes more informative, and they demonstrate strong empirical gains across vision, text, and tabular data with low computational overhead. This approach offers a lightweight, architecture-agnostic mechanism to safely incorporate external representations, including effective multi-source integration and resilience under noise and distribution shift, with implications for practical deployment in diverse real-world tasks.

Abstract

Transfer learning typically leverages representations learned from a source domain to improve performance on a target task. A common approach is to extract features from a pre-trained model and directly apply them for target prediction. However, this strategy is prone to negative transfer where the source representation fails to align with the target distribution. In this article, we propose Residual Feature Integration (REFINE), a simple yet effective method designed to mitigate negative transfer. Our approach combines a fixed source-side representation with a trainable target-side encoder and fits a shallow neural network on the resulting joint representation, which adapts to the target domain while preserving transferable knowledge from the source domain. Theoretically, we prove that REFINE is sufficient to prevent negative transfer under mild conditions, and derive the generalization bound demonstrating its theoretical benefit. Empirically, we show that REFINE consistently enhances performance across diverse application and data modalities including vision, text, and tabular data, and outperforms numerous alternative solutions. Our method is lightweight, architecture-agnostic, and robust, making it a valuable addition to the existing transfer learning toolbox.

Residual Feature Integration is Sufficient to Prevent Negative Transfer

TL;DR

Negative transfer from mismatched source representations limits transfer learning. ReFine remedies this by freezing , learning a target residual encoder , and training a shallow adapter on the joint features , ensuring performance is never worse than target-only training. The authors provide a risk bound showing a favorable trade-off between a parametric term and a nonparametric term that improves as the residual becomes more informative, and they demonstrate strong empirical gains across vision, text, and tabular data with low computational overhead. This approach offers a lightweight, architecture-agnostic mechanism to safely incorporate external representations, including effective multi-source integration and resilience under noise and distribution shift, with implications for practical deployment in diverse real-world tasks.

Abstract

Transfer learning typically leverages representations learned from a source domain to improve performance on a target task. A common approach is to extract features from a pre-trained model and directly apply them for target prediction. However, this strategy is prone to negative transfer where the source representation fails to align with the target distribution. In this article, we propose Residual Feature Integration (REFINE), a simple yet effective method designed to mitigate negative transfer. Our approach combines a fixed source-side representation with a trainable target-side encoder and fits a shallow neural network on the resulting joint representation, which adapts to the target domain while preserving transferable knowledge from the source domain. Theoretically, we prove that REFINE is sufficient to prevent negative transfer under mild conditions, and derive the generalization bound demonstrating its theoretical benefit. Empirically, we show that REFINE consistently enhances performance across diverse application and data modalities including vision, text, and tabular data, and outperforms numerous alternative solutions. Our method is lightweight, architecture-agnostic, and robust, making it a valuable addition to the existing transfer learning toolbox.
Paper Structure (26 sections, 5 theorems, 34 equations, 4 figures, 19 tables, 1 algorithm)

This paper contains 26 sections, 5 theorems, 34 equations, 4 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation and Algorithm
  4. Numerical Experiments
  5. Experiment setup
  6. Single-source transfer with noisy and imbalanced data
  7. Single-source transfer with distribution shift
  8. Multi-source transfer
  9. Theoretical Analysis
  10. Hölder class functions
  11. ReLU Neural Networks for Residual Connection
  12. Empirical Risk Minimization for ReFine
  13. Conclusion
  14. Proofs of Main Results
  15. Additional Notation
  16. ...and 11 more sections

Key Result

Theorem 5.1

Fix any external representation $f_\text{rep}: [0, 1]^d \to \mathcal{B}_p(1)$ and define the best linear probe Assume $\|v^*\| \leq 1$ and that the residual $f^* - v^{* \top} f_\text{rep}$ lies in the unit Hölder ball $\mathcal{C}^\beta_\text{u}$ with a non-integer $\beta > 0$. Let $\rho > 0$ be a tuning parameter (which serves as a proxy for the residual norm), and choose network parameters wher

Figures (4)

  • Figure 1: A schematic overview of ReFine.
  • Figure 2: Multi-source transfer of ReFine.
  • Figure 3: Performance of Adapter under varying parameter count multipliers.
  • Figure 4: ReFine Scaling with Clean Sources

Theorems & Definitions (10)

  • Theorem 5.1
  • proof : Proof Sketch of Theorem \ref{['thm: ERM ub']}
  • Remark 5.2
  • Remark 5.3
  • proof : Proof of Theorem \ref{['thm: ERM ub']}
  • Lemma B.1: Lemma 21 from nakada2020adaptive
  • Lemma B.2: Modification of Theorem 3.1 from petersen2018optimal
  • Lemma B.3: Modification to Lemma 4 from schmidt2020nonparametric
  • Lemma B.4
  • proof : Proof of Lemma \ref{['lem: covering number mN']}