Learning Multiple Tasks with Multilinear Relationship Networks

TL;DR

The paper tackles multi-task learning in deep networks by explicitly modeling relationships across tasks, features, and classes. It introduces Multilinear Relationship Networks (MRN), which place tensor normal priors on task-specific layer parameters to learn high-order, multilinear relationships via Kronecker-structured covariances and a MAP training objective. An alternating optimization strategy updates network weights and covariance factors, enabling joint learning of transferable features and task relationships. Empirical results on three benchmarks show MRN achieving state-of-the-art accuracy, with visualization confirming meaningful task covariances and improved feature transferability. The approach offers a flexible, scalable framework that generalizes across backbones and task settings, addressing both negative-transfer in feature layers and under-transfer in classifiers.

Abstract

Deep networks trained on large-scale data can learn transferable features to promote learning multiple tasks. Since deep features eventually transition from general to specific along deep networks, a fundamental problem of multi-task learning is how to exploit the task relatedness underlying parameter tensors and improve feature transferability in the multiple task-specific layers. This paper presents Multilinear Relationship Networks (MRN) that discover the task relationships based on novel tensor normal priors over parameter tensors of multiple task-specific layers in deep convolutional networks. By jointly learning transferable features and multilinear relationships of tasks and features, MRN is able to alleviate the dilemma of negative-transfer in the feature layers and under-transfer in the classifier layer. Experiments show that MRN yields state-of-the-art results on three multi-task learning datasets.

Figures (3)

• Figure 1: Multilinear relationship network (MRN) for multi-task learning: (1) convolutional layers $conv1$--$conv5$ and fully-connected layer $fc6$ learn transferable features, so their parameters are shared across tasks; (2) fully-connected layers $fc7$--$fc8$ fit task-specific structures, so their parameters are modeled by tensor normal priors for learning multilinear relationships of features, classes and tasks.
• Figure 2: Examples of the Office-Home dataset.
• Figure 3: Hinton diagram of task relationships (a)(b) and t-SNE embedding of deep features (c)(d).