HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models

TL;DR

A significant advance toward more flexible and comprehensive model merging techniques is marked by modeling the architecture-space merging process as a reinforcement learning task, and a multi-objective optimization paradigm is introduced to accommodate users' diverse task preferences.

Abstract

Model merging is a technique that combines multiple large pretrained models into a single model with enhanced performance and broader task adaptability. It has gained popularity in large pretrained model development due to its ability to bypass the need for original training data and further training processes. However, most existing model merging approaches focus solely on exploring the parameter space, merging models with identical architectures. Merging within the architecture space, despite its potential, remains in its early stages due to the vast search space and the challenges of layer compatibility. This paper marks a significant advance toward more flexible and comprehensive model merging techniques by modeling the architecture-space merging process as a reinforcement learning task. We train policy and value networks using offline sampling of weight vectors, which are then employed for the online optimization of merging strategies. Moreover, a multi-objective optimization paradigm is introduced to accommodate users' diverse task preferences, learning the Pareto front of optimal models to offer customized merging suggestions. Experimental results across multiple tasks, including text translation, mathematical reasoning, and code generation, validate the effectiveness and superiority of the proposed framework in model merging. The code will be made publicly available after the review process.

Figures (5)

• Figure 1: Illustration of model merge and our proposed method. (a) illustrates the concept and process of model merging. (b) presents the mathematical representation of model merging named task vector. Based on the task vector, model merging is performed through task arithmetic. (c) and (d) demonstrate the model merging process of TM3 at the parameter and architecture levels, respectively. In (c), HM3 uniformly generates parameter vectors for different base models, and then optimizes the optimal parameters using a combination of the DARE method and Ties Merging. In (d), based on the model obtained in (c), HM3 employs a reinforcement learning strategy to reconstruct the model within a layer-granularity search space.
• Figure 2: Metrics in merged models for different model merging methods.
• Figure 3: HV in different episodes for HM3.
• Figure 4: The convergence of RL in the HM3 at the architecture space.
• Figure 5: The convergence of RL in the HM3 at the architecture space.