Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries

TL;DR

This paper proposes an optimisation strategy that includes additional cloned tasks named self-auxiliaries into the learning process to flexibly transfer knowledge between tasks asymmetrically, and demonstrates that asymmetric knowledge transfer provides substantial improvements in performance compared to existing multi-task optimisation strategies on benchmark computer vision problems.

Abstract

Knowledge transfer in multi-task learning is typically viewed as a dichotomy; positive transfer, which improves the performance of all tasks, or negative transfer, which hinders the performance of all tasks. In this paper, we investigate the understudied problem of asymmetric task relationships, where knowledge transfer aids the learning of certain tasks while hindering the learning of others. We propose an optimisation strategy that includes additional cloned tasks named self-auxiliaries into the learning process to flexibly transfer knowledge between tasks asymmetrically. Our method can exploit asymmetric task relationships, benefiting from the positive transfer component while avoiding the negative transfer component. We demonstrate that asymmetric knowledge transfer provides substantial improvements in performance compared to existing multi-task optimisation strategies on benchmark computer vision problems.

Figures (3)

• Figure 1: Performance of typical learning setup vs. asymmetric learning setup using pre-computed self-auxiliaries for the NYUv2 dataset with varied number of shared layers, from no sharing (single-task learning models) to a fully shared encoder (shared bottom model).
• Figure 2: Conceptual diagram for a self-auxiliary task, in a branching multi-task architecture with two tasks. These tasks have an asymmetric relationship where $\mathcal{T}_1$ improves the learning of $\mathcal{T}_2$ but $\mathcal{T}_2$ impairs the learning of $\mathcal{T}_1$. To enable asymmetric transfer, a self-auxiliary $\mathcal{T}_{1 \to 2}$ is added. It is identical to its source task $\mathcal{T}_1$ and uses the modules of its target task $\mathcal{T}_2$ to share knowledge ($\mathcal{T}_{1 \to 2} \leftrightarrow \mathcal{T}_2$), and is discarded at inference. This induces positive transfer in task-specific layers of the $\mathcal{T}_2$ model ($\mathcal{T}_1 \to \mathcal{T}_2$) while avoiding negative transfer in task-specific layers of the $\mathcal{T}_1$ model ($\mathcal{T}_2 \not\to \mathcal{T}_1$).
• Figure 3: Visualisation of task relationships for computer vision datasets.