Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoD: A Distribution-Based Approach for Merging Large Language Models

Quy-Anh Dang, Chris Ngo

TL;DR

The MoD framework is proposed, a novel approach for merging LLMs that operates directly on their output probability distributions, rather than on model weights, which effectively preserves the specialized capabilities of individual models while enabling efficient knowledge sharing across tasks.

Abstract

Large language models (LLMs) have enabled the development of numerous specialized, task-specific variants. However, the maintenance and deployment of these individual models present substantial challenges in terms of resource utilization and operational efficiency. In this work, we propose the \textit{Mixture of Distributions (MoD)} framework, a novel approach for merging LLMs that operates directly on their output probability distributions, rather than on model weights. Unlike traditional weight-averaging methods, MoD effectively preserves the specialized capabilities of individual models while enabling efficient knowledge sharing across tasks. Through extensive experimentation on mathematical reasoning benchmarks using Qwen2.5 models, we demonstrate that MoD significantly outperforms existing model merging techniques across multiple benchmarks. All code, data, and experimental materials are published at https://github.com/knovel-eng/mod.

MoD: A Distribution-Based Approach for Merging Large Language Models

TL;DR

The MoD framework is proposed, a novel approach for merging LLMs that operates directly on their output probability distributions, rather than on model weights, which effectively preserves the specialized capabilities of individual models while enabling efficient knowledge sharing across tasks.

Abstract

Large language models (LLMs) have enabled the development of numerous specialized, task-specific variants. However, the maintenance and deployment of these individual models present substantial challenges in terms of resource utilization and operational efficiency. In this work, we propose the \textit{Mixture of Distributions (MoD)} framework, a novel approach for merging LLMs that operates directly on their output probability distributions, rather than on model weights. Unlike traditional weight-averaging methods, MoD effectively preserves the specialized capabilities of individual models while enabling efficient knowledge sharing across tasks. Through extensive experimentation on mathematical reasoning benchmarks using Qwen2.5 models, we demonstrate that MoD significantly outperforms existing model merging techniques across multiple benchmarks. All code, data, and experimental materials are published at https://github.com/knovel-eng/mod.

Paper Structure

This paper contains 18 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Merging
  4. Merging Techniques
  5. Methodology
  6. Notation and Symbols
  7. Motivation
  8. Mixture of Distributions (MoD)
  9. Solving for Mixture Weights
  10. Maximizing Density at Key Points
  11. Experiments
  12. Datasets
  13. Metrics
  14. Baseline Models
  15. Experimental Evaluation
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Comparison of our MoD method with the Weighted Average method. While weighted averaging methods for merging LLMs often produce new distributions that alter the characteristics of the original models (see Fig. \ref{['fig:wa-pipeline']}), our MoD approach effectively preserves key density structures, accurately maintaining peak densities at $\theta = 0$ and $\theta = 5$ (see Fig. \ref{['fig:main-pipeline']}).
  • Figure 2: Distribution distortion in weighted averaging methods, demonstrating failure to preserve maximum density at $\theta = 0$ despite high density in the red distribution.
  • Figure 3: MoD successfully preserves maximum density characteristics at $\theta = 0$, demonstrating effective distribution merging compared to traditional approaches.