Unconstrained Model Merging for Enhanced LLM Reasoning

TL;DR

This work proposes an unconstrained model merging framework that accommodates both homogeneous and heterogeneous model architectures with a focus on reasoning tasks, and reveals key findings that combinatorial reasoning emerges from merging which surpasses simple additive effects.

Abstract

Recent advancements in building domain-specific large language models (LLMs) have shown remarkable success, especially in tasks requiring reasoning abilities like logical inference over complex relationships and multi-step problem solving. However, creating a powerful all-in-one LLM remains challenging due to the need for proprietary data and vast computational resources. As a resource-friendly alternative, we explore the potential of merging multiple expert models into a single LLM. Existing studies on model merging mainly focus on generalist LLMs instead of domain experts, or the LLMs under the same architecture and size. In this work, we propose an unconstrained model merging framework that accommodates both homogeneous and heterogeneous model architectures with a focus on reasoning tasks. A fine-grained layer-wise weight merging strategy is designed for homogeneous models merging, while heterogeneous model merging is built upon the probabilistic distribution knowledge derived from instruction-response fine-tuning data. Across 7 benchmarks and 9 reasoning-optimized LLMs, we reveal key findings that combinatorial reasoning emerges from merging which surpasses simple additive effects. We propose that unconstrained model merging could serve as a foundation for decentralized LLMs, marking a notable progression from the existing centralized LLM framework. This evolution could enhance wider participation and stimulate additional advancement in the field of artificial intelligence, effectively addressing the constraints posed by centralized models.

Figures (9)

• Figure 1: The framework on unconstrained model merging. We first establish a robust evaluator and select the top-ranking domain-specific small models (DSSMs) with the strongest math or coding abilities. For unconstrained model merging, we apply the CMA-ES algorithm to search for optimal parameters for homogeneous models, and leverage instruction-response distribution based fusion for heterogeneous LLMs.
• Figure 2: The accuracy of the source models and fusion models in solving problems of different math levels.
• Figure 3: The accuracy of the source models and fusion models in solving problems of different math types.
• Figure 4: The performance of the fusion model on the MATH across different types and difficulty levels.
• Figure 5: The Performance of COT reasoning on MATH topics and difficulty levels.