MoCo: A One-Stop Shop for Model Collaboration Research

TL;DR

MoCo addresses fragmentation in model collaboration by introducing a unified library and benchmark suite that implements 26 collaboration algorithms across four information-exchange levels and 25 evaluation datasets. The framework enables fair, scalable comparison and reveals that collaboration improves performance in the majority of settings (61.0%), with notable gains up to 25.8% in certain domains. Key insights include the superior performance of text-level and weight-level methods, the positive impact of increasing model pool size and diversity, and the phenomenon of collaborative emergence where problems become solvable only in a collaborative setting. By providing extensible tooling and open data, MoCo aims to catalyze open, modular, and decentralized AI development and guide future research in compositional AI.

Abstract

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.

Figures (8)

• Figure 1: MoCo is a comprehensive library for model collaboration research. Download MoCo, write a config file specifying model collaboration setups (models, data, hardware, etc.), execute and compare diverse model collaboration algorithms with MoCo.
• Figure 2: Scaling the number of models in model collaboration systems and evaluating on reasoning, QA, and safety domains. We observe a consistent upward trend that further improves over the best single model, with text-level and weight-level methods being more scalable and benefiting from a larger pool of diverse models. This indicates that by scaling up model collaboration, we could build bottom-up compositional AI systems where the components are small but the system is large.
• Figure 3: Impact of model pool diversity on collaboration performance. The x-axis shows the configurations of model pool diversity: $1\times 8, 2 \times 4, 4 \times 2$ and $8 \times 1$. Results demonstrate that model collaboration benefits from increased diversity among participating models, indicating the need for model specialization.
• Figure 4: For problems where none of the LLMs could solve individually, what percentage of them are solvable with the model collaboration system, across diverse tasks and collaboration strategies. We observe consistent collaborative emergence across settings with an average of 18.5%, indicating that many model collaboration algorithms do not merely offer a union of existing capabilities: new skills emerge in the collaborative system of multiple models that solve problems where individual models struggle to.
• Figure 5: Employing random, prompt-based, or description-based strategies to select 3 models out of 8 for collaboration. Both strategies outperform the random baseline and no collaboration, indicating the importance of model selection strategies and highlighting the need for future research.