Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Talk Structurally, Act Hierarchically: A Collaborative Framework for LLM Multi-Agent Systems

Zhao Wang, Sota Moriyama, Wei-Yao Wang, Briti Gangopadhyay, Shingo Takamatsu

TL;DR

TalkHier presents a dynamic graph-based LLM-MA framework that combines a structured, context-rich communication protocol with hierarchical refinement to address disorganized exchanges, memory bottlenecks, and biased feedback. By deploying independent agent memories and a supervisor-driven task routing mechanism, TalkHier achieves state-of-the-art performance across MMLU, WikiQA, and ad-text generation benchmarks, outperforming both single-agent baselines and existing multi-agent systems. Ablation studies confirm the necessity of the evaluation hierarchy, independent memories, and context-rich communication for robust performance. While the results are compelling, the approach incurs higher API costs, pointing to future work on efficiency and accessibility without sacrificing collaboration quality.

Abstract

Recent advancements in LLM-based multi-agent (LLM-MA) systems have shown promise, yet significant challenges remain in managing communication and refinement when agents collaborate on complex tasks. In this paper, we propose \textit{Talk Structurally, Act Hierarchically (TalkHier)}, a novel framework that introduces a structured communication protocol for context-rich exchanges and a hierarchical refinement system to address issues such as incorrect outputs, falsehoods, and biases. \textit{TalkHier} surpasses various types of SoTA, including inference scaling model (OpenAI-o1), open-source multi-agent models (e.g., AgentVerse), and majority voting strategies on current LLM and single-agent baselines (e.g., ReAct, GPT4o), across diverse tasks, including open-domain question answering, domain-specific selective questioning, and practical advertisement text generation. These results highlight its potential to set a new standard for LLM-MA systems, paving the way for more effective, adaptable, and collaborative multi-agent frameworks. The code is available https://github.com/sony/talkhier.

Talk Structurally, Act Hierarchically: A Collaborative Framework for LLM Multi-Agent Systems

TL;DR

TalkHier presents a dynamic graph-based LLM-MA framework that combines a structured, context-rich communication protocol with hierarchical refinement to address disorganized exchanges, memory bottlenecks, and biased feedback. By deploying independent agent memories and a supervisor-driven task routing mechanism, TalkHier achieves state-of-the-art performance across MMLU, WikiQA, and ad-text generation benchmarks, outperforming both single-agent baselines and existing multi-agent systems. Ablation studies confirm the necessity of the evaluation hierarchy, independent memories, and context-rich communication for robust performance. While the results are compelling, the approach incurs higher API costs, pointing to future work on efficiency and accessibility without sacrificing collaboration quality.

Abstract

Recent advancements in LLM-based multi-agent (LLM-MA) systems have shown promise, yet significant challenges remain in managing communication and refinement when agents collaborate on complex tasks. In this paper, we propose \textit{Talk Structurally, Act Hierarchically (TalkHier)}, a novel framework that introduces a structured communication protocol for context-rich exchanges and a hierarchical refinement system to address issues such as incorrect outputs, falsehoods, and biases. \textit{TalkHier} surpasses various types of SoTA, including inference scaling model (OpenAI-o1), open-source multi-agent models (e.g., AgentVerse), and majority voting strategies on current LLM and single-agent baselines (e.g., ReAct, GPT4o), across diverse tasks, including open-domain question answering, domain-specific selective questioning, and practical advertisement text generation. These results highlight its potential to set a new standard for LLM-MA systems, paving the way for more effective, adaptable, and collaborative multi-agent frameworks. The code is available https://github.com/sony/talkhier.

Paper Structure

This paper contains 60 sections, 8 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Collaborative LLM-MA.
  4. Communication in LLM-MA.
  5. Feedback-Based Refinement.
  6. Methodology
  7. Agents with Independent Memory
  8. Agent-Specific Memory.
  9. Context-Rich Communication Between Agents
  10. Communication Event Sequence.
  11. Collaborative Hierarchy Agent Team
  12. Algorithm.
  13. Experiments
  14. Experimental Setup
  15. Datasets.
  16. ...and 45 more sections

Figures (5)

  • Figure 1: Existing LLM-MA methods (left) face two major challenges: 1) disorganized, lengthy text-based communication protocols, and 2) sequential or overly similar flat multi-agent refinements. In contrast, TalkHier (right) introduces a well-structured communication protocol and a hierarchical refinement approach.
  • Figure 2: Our TalkHier built on GPT4o surpasses inference scaling models (OpenAI-o1), open-source multi-agent models (AgentVerse and etc.), and models with majority voting strategies (ReAct, GPT4o) on five subtasks of MMLU.
  • Figure 3: Comparisons between existing approaches (left) and ours (right). Our TalkHier proposes a new communication protocol (first row) featuring context-rich and well-structured communication information, along with a collaborative hierarchical refinement (second row) where evaluations provide summarized and coordinated feedback within an LLM-MA framework.
  • Figure 4: Prompts for acquiring the contents of the context-rich, structured communication protocol in TalkHier.
  • Figure 5: Illustrated hierarchy of TalkHier.