Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cascaded Transfer: Learning Many Tasks under Budget Constraints

Eloi Campagne, Yvenn Amara-Ouali, Yannig Goude, Mathilde Mougeot, Argyris Kalogeratos

TL;DR

The Cascaded Transfer Learning is introduced, a novel many-task transfer learning paradigm where information cascades hierarchically through tasks that are learned by individual models of the same class, while respecting given budget constraints.

Abstract

Many-Task Learning refers to the setting where a large number of related tasks need to be learned, the exact relationships between tasks are not known. We introduce the Cascaded Transfer Learning, a novel many-task transfer learning paradigm where information (e.g. model parameters) cascades hierarchically through tasks that are learned by individual models of the same class, while respecting given budget constraints. The cascade is organized as a rooted tree that specifies the order in which tasks are learned and refined. We design a cascaded transfer mechanism deployed over a minimum spanning tree structure that connects the tasks according to a suitable distance measure, and allocates the available training budget along its branches. Experiments on synthetic and real many-task settings show that the resulting method enables more accurate and cost effective adaptation across large task collections compared to alternative approaches.

Cascaded Transfer: Learning Many Tasks under Budget Constraints

TL;DR

The Cascaded Transfer Learning is introduced, a novel many-task transfer learning paradigm where information cascades hierarchically through tasks that are learned by individual models of the same class, while respecting given budget constraints.

Abstract

Many-Task Learning refers to the setting where a large number of related tasks need to be learned, the exact relationships between tasks are not known. We introduce the Cascaded Transfer Learning, a novel many-task transfer learning paradigm where information (e.g. model parameters) cascades hierarchically through tasks that are learned by individual models of the same class, while respecting given budget constraints. The cascade is organized as a rooted tree that specifies the order in which tasks are learned and refined. We design a cascaded transfer mechanism deployed over a minimum spanning tree structure that connects the tasks according to a suitable distance measure, and allocates the available training budget along its branches. Experiments on synthetic and real many-task settings show that the resulting method enables more accurate and cost effective adaptation across large task collections compared to alternative approaches.
Paper Structure (42 sections, 3 theorems, 25 equations, 8 figures, 3 tables, 1 algorithm)

This paper contains 42 sections, 3 theorems, 25 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related applications.
  3. Contributions.
  4. Cascaded Transfer Learning
  5. Preliminaries
  6. The Cascaded Transfer Learning Paradigm
  7. Cascade Construction Strategies.
  8. Default Tree Construction.
  9. Computational Complexity.
  10. Extensions.
  11. Theoretical Analysis
  12. Notations.
  13. Parameter-Space Analysis
  14. CTL vs TL. Why Cascades?
  15. Feature-Space Analysis
  16. ...and 27 more sections

Key Result

Proposition 3.1

For any task $v \neq v_0$, and a path $(v_0\!\to\!v_1\!\to\!\cdots\!\to\!v_m\!=\!v$), involving $m >1$ intermediate nodes, it holds: where $P_{i:m} =\prod_{j=i}^m\rho_{v_j}^{b_{v_j}}$ is a multiplicative attenuation factor.

Figures (8)

  • Figure 1: Parameter-space intuition for CTL. Arrows are shown along learning trajectories associated to different tasks (i.e. iterative parameter optimization). Each learning trajectory has a distinct color and stops at a learned model (solid nodes) that is short of its optimum (white-filled nodes) due to the limited available training budget. Left: Independent training. Each task is optimized from its own initialization. Middle: Star transfer. One source task is learned first and directly transferred to all other tasks. Right: CTL where tasks are learned sequentially along a minimum spanning tree. Long transfers are realized by a series steps involving intermediate tasks.
  • Figure 2: Experimental protocol. The dataset of each task $v_i$ is split into train/test sets. Training-phase components (distance computation, tree construction, cascaded transfer) operate solely on training data. Final evaluation uses held-out test sets, with results averaged over $50$ random seeds.
  • Figure 3: Radar plot of average improvement (%) as compared to the Star baseline, across datasets and budgets.
  • Figure 4: Distance taxonomy for MST-based cascade construction.
  • Figure 5: Two-dimensional PCA projection of task parameters for the synthetic data with increasing within-cluster variance $\tau_{\text{within}}$.
  • ...and 3 more figures

Theorems & Definitions (13)

  • Definition 2.1
  • Proposition 3.1: Cascaded transfer over a path
  • Theorem 3.2
  • Proposition 3.3: Expected noisy propagation along a path
  • proof : Cascaded transfer over a path
  • proof : Path-wise error decomposition
  • proof : Feature-space contraction
  • proof : Edge-wise propagation in feature space
  • proof : Path-wise error propagation in feature space
  • proof : Empirical optimum
  • ...and 3 more