Efficient Multi-Task Modeling through Automated Fusion of Trained Models

TL;DR

The paper addresses the inefficiency of traditional multi-task learning (MTL) that relies on task-specific design choices. It introduces Efficient Multi-Task Modeling (EMM), which automatically fuses trained single-task models by decomposing them into hierarchical components and applying an Adaptive Knowledge Fusion (AKF) module that performs intra-task MoE-based fusion and inter-task MTM via self-attention. Through stacking AKF modules, EMM constructs a flexible, automated multi-task model that accommodates heterogeneous architectures and task relationships. Experimental results on Census-Income, Ali-CCP, and AliExpress demonstrate that EMM achieves strong or superior performance compared to baselines while reducing manual design effort, highlighting its practical value for scalable, automated multi-task modeling.

Abstract

Although multi-task learning is widely applied in intelligent services, traditional multi-task modeling methods often require customized designs based on specific task combinations, resulting in a cumbersome modeling process. Inspired by the rapid development and excellent performance of single-task models, this paper proposes an efficient multi-task modeling method that can automatically fuse trained single-task models with different structures and tasks to form a multi-task model. As a general framework, this method allows modelers to simply prepare trained models for the required tasks, simplifying the modeling process while fully utilizing the knowledge contained in the trained models. This eliminates the need for excessive focus on task relationships and model structure design. To achieve this goal, we consider the structural differences among various trained models and employ model decomposition techniques to hierarchically decompose them into multiple operable model components. Furthermore, we have designed an Adaptive Knowledge Fusion (AKF) module based on Transformer, which adaptively integrates intra-task and inter-task knowledge based on model components. Through the proposed method, we achieve efficient and automated construction of multi-task models, and its effectiveness is verified through extensive experiments on three datasets.

Figures (3)

• Figure 1: Illustrates Efficient Multi-Task Modeling (EMM) framework. When presenting the model pool and model structures, we employ different colors to distinguish various tasks and utilize diverse model styles to represent various model structures. Specifically, model structures with the same background color indicate that this part of the model shares the same input and output characteristics. The core advantage of the EMM method lies in its ability to deconstruct trained models into multiple independent model components and combine them hierarchically using the AKF module. For the same task, this method employs the MoE ideology to achieve knowledge fusion; whereas for different tasks, it adopts the Multi-Task Merge (MTM) method for fusion after adaptive screening. Subsequently, by stacking these component layers, we can effectively achieve multi-task modeling.
• Figure 2: The process of knowledge fusion is described. The task gating network is responsible for merging the knowledge from model components of the same task and the same hierarchical level. The fusion gating network selects model components from different tasks but at the same hierarchical level. After the selection is completed, the Multi-Task Merge (MTM) method is used to integrate the knowledge from components originating from different tasks.
• Figure 3: To test the adaptability of the EMM method for multi-task processing, we increased the number of tasks. In the experimental results charts, the X-axis represents the different tasks, while the Y-axis shows the AUC evaluation values.