The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Models

TL;DR

This work introduces the Law of Multi-model Collaboration, a scaling-law framework that predicts the theoretical performance limits of ensembles of large language models (LLMs) as a function of the aggregated parameter budget. By adopting a method-agnostic, oracle-based formulation, the authors quantify the intrinsic lower bound on cross-entropy loss, showing that multi-model ensembles follow a power-law scaling with respect to $P$, the total parameter count, and achieve a lower asymptotic loss floor than any single model. A key finding is that heterogeneous ensembles—combining models from different families—exhibit larger scaling exponents $\alpha$ and lower loss floors $\mathcal{L}_{\infty}$, indicating that diversity in inductive biases drives substantial gains. The study uses a comprehensive empirical protocol over 71 open-source bases, revealing that distributing parameters across diverse models can be more effective than monolithic scaling, thereby reframing ensemble methods as principled, diversity-aware scaling strategies for advancing the intelligence frontier.

Abstract

Recent advances in large language models (LLMs) have been largely driven by scaling laws for individual models, which predict performance improvements as model parameters and data volume increase. However, the capabilities of any single LLM are inherently bounded. One solution originates from intricate interactions among multiple LLMs, rendering their collective performance surpasses that of any constituent model. Despite the rapid proliferation of multi-model integration techniques such as model routing and post-hoc ensembling, a unifying theoretical framework of performance scaling for multi-model collaboration remains absent. In this work, we propose the Law of Multi-model Collaboration, a scaling law that predicts the performance limits of LLM ensembles based on their aggregated parameter budget. To quantify the intrinsic upper bound of multi-model collaboration, we adopt a method-agnostic formulation and assume an idealized integration oracle where the total cross-entropy loss of each sample is determined by the minimum loss of any model in the model pool. Experimental results reveal that multi-model systems follow a power-law scaling with respect to the total parameter count, exhibiting a more significant improvement trend and a lower theoretical loss floor compared to single model scaling. Moreover, ensembles of heterogeneous model families achieve better performance scaling than those formed within a single model family, indicating that model diversity is a primary driver of collaboration gains. These findings suggest that model collaboration represents a critical axis for extending the intelligence frontier of LLMs.

Figures (2)

• Figure 1: Comparison between scaling performance under two settings: single model and multi-model ensemble. Solid curves denote nonlinear least-squares fits of the scaling formula, while points correspond to Pareto-optimal lower envelopes. Model ensembling requires fewer parameters to achieve a target loss, and reduces the lower bound of the test loss significantly.
• Figure 2: Comparison between scaling performance between model ensemble within the same family and that across different families. Solid curves denote nonlinear least-squares fits of the scaling formula, while points correspond to Pareto-optimal lower envelopes. Collaboration beyond a single model series results in better scaling performance, validating the complementarity among different model series.