SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents

TL;DR

SMoA tackles inefficiency and limited diversity in dense multi-agent LLMs by introducing sparse information routing via a Judge and a Moderator, coupled with role-based diversity among agents. It extends the mixture-of-agents paradigm with sparse selection and dynamic stopping to balance performance and compute. Through extensive evaluation across alignment, reasoning, and fairness benchmarks using open- and closed-source models, SMoA achieves competitive results to MoA at substantially lower cost and with greater stability. The findings indicate strong scalability potential and show that hyper-parameter tuning can further enhance efficiency and capability.

Abstract

While multi-agent systems have been shown to significantly enhance the performance of Large Language Models (LLMs) across various tasks and applications, the dense interaction between scaling agents potentially hampers their efficiency and diversity. To address these challenges, we draw inspiration from the sparse mixture-of-agents (SMoE) and propose a sparse mixture-of-agents (SMoA) framework to improve the efficiency and diversity of multi-agent LLMs. Unlike completely connected structures, SMoA introduces novel Response Selection and Early Stopping mechanisms to sparsify information flows among individual LLM agents, striking a balance between performance and efficiency. Additionally, inspired by the expert diversity principle in SMoE frameworks for workload balance between experts, we assign distinct role descriptions to each LLM agent, fostering diverse and divergent thinking. Extensive experiments on reasoning, alignment, and fairness benchmarks demonstrate that SMoA achieves performance comparable to traditional mixture-of-agents approaches but with significantly lower computational costs. Further analysis reveals that SMoA is more stable, has a greater capacity to scale, and offers considerable potential through hyper-parameter optimization. Code and data will be available at: https://github.com/David-Li0406/SMoA.

Figures (5)

• Figure 1: Comparison among the pipeline structures of MAD, MoA and SMoA.
• Figure 2: Our Sparse Mixture-of-Agent (SMoA) framework.
• Figure 3: (a) Performance comparison on Just-Eval and CEB; (b) Performance comparison on MMAU.
• Figure 4: Budget analysis result on MMAU.
• Figure 5: Scaling analysis result on code subset of MMAU.