Guided Collaboration in Heterogeneous LLM-Based Multi-Agent Systems via Entropy-Based Understanding Assessment and Experience Retrieval

TL;DR

Heterogeneous multi-agent systems with diverse LLMs often suffer from negative synergy when strong and weak agents collaborate, due to cognitive mismatch and overload. The authors introduce an Entropy-Based Adaptive Guidance framework that uses multi-dimensional entropy metrics Hu to assess weak agents, three-tier guidance levels, and dynamic thresholds, complemented by a Retrieval-Augmented Generation module to capture and reuse successful experiences. Across GSM8K, MBPP, and CVRP, the approach consistently improves performance and stability over No-Guidance and Chain-of-Thought baselines, with additional gains from RAG particularly on reasoning and routing tasks. The findings show that the weakest agent largely constrains overall performance, and that adaptive, state-aware guidance plus memory-based experience retention offers a practical and scalable route to robust heterogeneous collaboration in LLM-based AI systems.

Abstract

With recent breakthroughs in large language models (LLMs) for reasoning, planning, and complex task generation, artificial intelligence systems are transitioning from isolated single-agent architectures to multi-agent systems with collaborative intelligence. However, in heterogeneous multi-agent systems (HMAS), capability differences among agents give rise to consistent cognitive problems, where strong and weak models fail to contribute effectively. We define the collaboration as a strong-weak system. Through comprehensive experiments, we disclose a counterintuitive phenomenon in the strong-weak system: a strong-weak collaboration may under-perform weak-weak combinations, revealing that cognitive mismatching are key bottlenecks limiting heterogeneous cooperation. To overcome these challenges, we propose an Entropy-Based Adaptive Guidance Framework that dynamically aligns the guidance with the cognitive state of each agent. The framework quantifies the understanding of weak agents through multi-dimensional entropy metrics - covering expression, uncertainty, structure, coherence, and relevance - and adaptively adjusts the intensity of the guidance at light, moderate and intensive levels. Furthermore, a Retrieval-Augmented Generation (RAG) mechanism is incorporated to retain successful collaboration experiences, enabling both immediate adaptation and long-term learning. Extensive experiments on three benchmark datasets, GSM8K, MBPP, and CVRP demonstrate that our approach consistently enhances the effectiveness and stability of heterogeneous collaboration. The results highlight that adaptive guidance not only mitigates cognitive imbalance but also establishes a scalable pathway toward more robust, cooperative multi-agent intelligence.

Figures (7)

• Figure 1: Illustration of collaboration patterns across three agent configurations on a simple mathematical reasoning task. Top: Strong-Strong collaboration achieves correct answer (Area = 40) through efficient information transfer. Middle: Strong-Weak collaboration produces incorrect answer (Area = 25) despite receiving identical high-quality guidance, demonstrating information loss due to cognitive mismatch. Bottom: Weak-Weak collaboration achieves better performance (Area = 64) than Strong-Weak through matched comprehension levels, revealing the negative synergy effect in heterogeneous collaboration.
• Figure 2: Overall framework of the heterogeneous multi-agent collaboration process.
• Figure 3: Average performance of heterogeneous agent combinations (Strong-Strong, Weak-Weak, Strong-Weak) across three benchmark datasets (GSM8K, MBPP, CVRP). Each subplot corresponds to a dataset.
• Figure 4: Performance gains of our entropy-based adaptive guidance methods compared to No-Guidance and Chain-of-Thought baselines across three benchmark datasets.
• Figure 5: Comparative performance of four collaboration strategies (No-Guidance, Chain-of-Thought, Guided, and Guided+RAG) across three benchmark datasets. Results demonstrate consistent superiority of our entropy-based adaptive guidance approach, with Guided+RAG achieving the highest performance across all tasks.

Theorems & Definitions (1)

• Definition 1: Heterogeneous Multi-Agent System