Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis

TL;DR

Attention-MoA tackles the challenge of enabling effective inference-time collaboration across multiple agents by introducing inter-agent semantic attention and inter-layer residual connections. The approach allows explicit peer critique, self-refinement, and long-context preservation with an adaptive early stopping mechanism to manage compute. Across AlpacaEval 2.0, MT-Bench, and FLASK, Attention-MoA achieves state-of-the-art results and, in small open-source configurations, rivals or surpasses large proprietary models. This work demonstrates scalable, cost-efficient multi-agent collaboration suitable for open ecosystems.

Abstract

As the development of Large Language Models (LLMs) shifts from parameter scaling to inference-time collaboration, the Mixture-of-Agents (MoA) framework has emerged as a general paradigm to harness collective intelligence by layering diverse models. While recent MoA variants have introduced dynamic routing and residual connections to improve efficiency, these methods often fail to facilitate deep semantic interaction between agents, limiting the system's ability to actively correct hallucinations and refine logic. In this paper, we introduce Attention-MoA, a novel MoA-based framework that redefines collaboration through Inter-agent Semantic Attention. Complemented by an Inter-layer Residual Module with Adaptive Early Stopping Mechanism, our architecture mitigates information degradation in deep layers while improving computational efficiency. Extensive evaluations across AlpacaEval 2.0, MT-Bench, and FLASK demonstrate that Attention-MoA significantly outperforms state-of-the-art baselines, achieving a 91.15% Length-Controlled Win Rate on AlpacaEval 2.0 and dominating in 10 out of 12 capabilities on FLASK. Notably, Attention-MoA enables an ensemble of small open-source models to outperform massive proprietary models like Claude-4.5-Sonnet and GPT-4.1, achieving an MT-Bench score of 8.83 and an AlpacaEval 2.0 LC Win Rate of 77.36%.

Figures (12)

• Figure 1: Overview of the Attention-MoA Framework. (a) The intra-layer workflow where heterogeneous collaborative agents ($\mathcal{A}^{col}$) generate responses refined by inter-agent semantic attention mechanism and Intra-layer Summary Agent ($\mathcal{A}^{sum}$). (b) The inter-layer residual pathway where residual agents ($\mathcal{A}^{res}$) incorporate an adaptive early stopping mechanism to dynamically control inference depth.
• Figure 2: Fine-grained evaluation on the FLASK dataset (Hard subset). Subfigure (a) demonstrates the performance gain of Attention-MoA over individual constituent models, while (b) highlights the improvements compared to other MoA-based architectures.
• Figure 3: Category-wise performance breakdown on MT-Bench. Subfigure (a) illustrates the comparison with individual models, while (b) shows the comparison against MoA-based baselines.
• Figure 4: Impact of the number of Collaborative Agents and the capability of the Aggregation Agent on AlpacaEval 2.0 LC Win Rate. The x-axis represents the set of collaborative agents, increasing in size from left to right.
• Figure 5: Performance comparison with other MoA-based baselines on AlpacaEval 2.0 across different layer depths.