Characterization of Transfer Using Multi-task Learning Curves

TL;DR

This paper tackles the challenge of characterizing transfer effects in multi-task learning, especially under highly incomplete data common in drug–target interaction (DTI) problems. It introduces multi-task learning curves (MTLCs), a parametric framework (including EXP3 and its extensions) that models inductive performance as a function of sample size, with separate components for contextual and pairwise transfer. An efficient grid-based estimation method, statTAG, enables robust fitting of MTLCs to real-world DTI data (KiBA244), revealing domain-wide and pairwise transfer patterns and showing limitations of gradient-based transfer methods in this setting. The work suggests MTLCs as a powerful tool for transfer analysis and active learning in domains with sparse, missing-not-at-random data, with implications for foundation models and large-scale multi-task orchestration.

Abstract

Transfer effects manifest themselves both during training using a fixed data set and in inductive inference using accumulating data. We hypothesize that perturbing the data set by including more samples, instead of perturbing the model by gradient updates, provides a complementary and more fundamental characterization of transfer effects. To capture this phenomenon, we quantitatively model transfer effects using multi-task learning curves approximating the inductive performance over varying sample sizes. We describe an efficient method to approximate multi-task learning curves analogous to the Task Affinity Grouping method applied during training. We compare the statistical and computational approaches to transfer, which indicates considerably higher compute costs for the previous but better power and broader applicability. Evaluations are performed using a benchmark drug-target interaction data set. Our results show that learning curves can better capture the effects of multi-task learning and their multi-task extensions can delineate pairwise and contextual transfer effects in foundation models.

Figures (6)

• Figure 1: Overview.(A) Data: The main data set contains 50k samples with 32k complete input descriptors and 244 output tasks, which is highly incomplete (90%< is missing). Cross-validation folds are generated at the sample and perturbed sample group (scaffold) levels. (B) Model: The 244 tasks correspond to 244 head nodes on a shared foundation layer with 2k nodes in a multiple-output MLP. (C) Transfer: Transfer learning can change the effective sample size, learning rate, and asymptotic limit of a target task. (D) Learning curves: The effects of sample sizes for the target (horizontal axis), the auxiliary (illustrated at 0 and $n^\mathrm{max}_\mathrm{aux}$) and the complementary (illustrated at 0 and $n^\mathrm{max}_{\Sigma}$) tasks are modeled in both single and multi-task learning (STL/MTL) scenarios using 1/2/3-argument learning curves (LC, MTLC.2/3). (D-E) Computational transfer: Transfer effects using cross-task training versus cross-task samples are analyzed and compared for systematically varying sample sizes.
• Figure 2: Comparison of the generalization performance estimations over the compounds versus over the scaffolds in the case of the KIBA data set using $1$ fold for training and the complementary data for testing.
• Figure 3: Task-by-task comparison of AUROC and AUPR performances in single-task and multi-task learning trained on $9$ folds of the KIBA244 data set, validated on the tenth fold; values averaged over the $10$ training-validation settings.
• Figure 4: Scatter plot showing the relation of EXP3.1 parameters between ST and MT learning curves for performance measures AUROC and AUPR.
• Figure 5: Scatter plot showing the relation between differences in performance measure AUROC and AUPR and differences in parameter $c_i$.