Knowledge Fusion of Chat LLMs: A Preliminary Technical Report
Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, Wei Bi
TL;DR
FusionChat addresses the challenge of integrating knowledge from multiple chat LLMs with diverse architectures by externalizing each model's knowledge as probabilistic distribution matrices and applying a two-stage process: (i) pairwise knowledge fusion to produce identically structured target LLMs via lightweight fine-tuning, and (ii) parameter-space merging using Variation Ratio Merge (VaRM) to allocate per-parameter merging weights. The VaRM mechanism computes weights as $W_{j,m} = \frac{\mathbb{E}_{m}\Delta\theta^{2}_{j,m}}{\sum^{K-1}_{j=1}\mathbb{E}_{m}\Delta\theta^{2}_{j,m}}$, enabling fine-grained integration of updates across matrices. The approach is validated on three open-source chat LLMs, achieving FusionChat-7B scores that surpass the corresponding fine-tuned baselines and closely approach Mixtral-level performance on MT-Bench, while offering plug-and-play scalability for new sources and improved memory efficiency during inference. Overall, FusionChat demonstrates that a fuse-then-merge paradigm with VaRM can effectively fuse heterogeneous chat models into a single, capable LLM with practical deployment advantages.
Abstract
Recently, FuseLLM introduced the concept of knowledge fusion to transfer the collective knowledge of multiple structurally varied LLMs into a target LLM through lightweight continual training. In this report, we extend the scalability and flexibility of the FuseLLM framework to realize the fusion of chat LLMs, resulting in FusionChat. FusionChat comprises two main stages. Firstly, we undertake knowledge fusion for structurally and scale-varied source LLMs to derive multiple target LLMs of identical structure and size via lightweight fine-tuning. Then, these target LLMs are merged within the parameter space, wherein we propose a novel method for determining the merging weights based on the variation ratio of parameter matrices before and after fine-tuning. We validate our approach using three prominent chat LLMs with diverse architectures and scales, namely NH2-Mixtral-8x7B, NH2-Solar-10.7B, and OpenChat-3.5-7B. Experimental results spanning various chat domains demonstrate the superiority of FusionChat-7B across a broad spectrum of chat LLMs at 7B and 34B scales, even surpassing GPT-3.5 (March) and approaching Mixtral-8x7B-Instruct.