Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning

Zhenyu Bi, Meng Lu, Yang Li, Swastik Roy, Weijie Guan, Morteza Ziyadi, Xuan Wang

TL;DR

OptAgent introduces a graph-based, verbal reinforcement learning framework that jointly optimizes multi-agent interactions among LLMs to improve reasoning. By employing two meta-agents for reflection and action, it dynamically builds and refines communication structures, emphasizing interaction quality alongside agent performance. Across math, science, creative writing, and sorting tasks, OptAgent outperforms single-agent prompting and several state-of-the-art multi-agent baselines, with the strongest gains seen in 5-agent configurations. The approach highlights the importance of task-specific communication schemas and raises considerations about computational cost and backbone-model dependencies in multi-agent LLM systems.

Abstract

Large Language Models (LLMs) have shown remarkable reasoning capabilities in mathematical and scientific tasks. To enhance complex reasoning, multi-agent systems have been proposed to harness the collective intelligence of LLM agents. However, existing collaboration structures are either predefined or rely on majority voting or round-table debates, which can suppress correct but less dominant agent contributions. Recent approaches model multi-agent systems as graph networks but optimize purely for agent performance, neglecting the quality of interactions. We hypothesize that effective agent communication is crucial for multi-agent reasoning and that debating quality plays a significant role. To address this, we propose $\ours$, a multi-agent verbal reinforcement learning algorithm that dynamically constructs and refines multi-agent collaboration structures. Our method defines action spaces and a feedback mechanism that evaluates communication robustness and coherence throughout the debate. The final decision is achieved through a majority vote over all the agents. We assess $\ours$ on various reasoning tasks, including mathematical reasoning, creative writing, scientific reasoning, and numerical sorting. Results demonstrate that our approach significantly outperforms single-agent prompting methods and state-of-the-art multi-agent frameworks on diverse tasks.

OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning

TL;DR

OptAgent introduces a graph-based, verbal reinforcement learning framework that jointly optimizes multi-agent interactions among LLMs to improve reasoning. By employing two meta-agents for reflection and action, it dynamically builds and refines communication structures, emphasizing interaction quality alongside agent performance. Across math, science, creative writing, and sorting tasks, OptAgent outperforms single-agent prompting and several state-of-the-art multi-agent baselines, with the strongest gains seen in 5-agent configurations. The approach highlights the importance of task-specific communication schemas and raises considerations about computational cost and backbone-model dependencies in multi-agent LLM systems.

Abstract

Large Language Models (LLMs) have shown remarkable reasoning capabilities in mathematical and scientific tasks. To enhance complex reasoning, multi-agent systems have been proposed to harness the collective intelligence of LLM agents. However, existing collaboration structures are either predefined or rely on majority voting or round-table debates, which can suppress correct but less dominant agent contributions. Recent approaches model multi-agent systems as graph networks but optimize purely for agent performance, neglecting the quality of interactions. We hypothesize that effective agent communication is crucial for multi-agent reasoning and that debating quality plays a significant role. To address this, we propose , a multi-agent verbal reinforcement learning algorithm that dynamically constructs and refines multi-agent collaboration structures. Our method defines action spaces and a feedback mechanism that evaluates communication robustness and coherence throughout the debate. The final decision is achieved through a majority vote over all the agents. We assess on various reasoning tasks, including mathematical reasoning, creative writing, scientific reasoning, and numerical sorting. Results demonstrate that our approach significantly outperforms single-agent prompting methods and state-of-the-art multi-agent frameworks on diverse tasks.
Paper Structure (74 sections, 5 figures, 9 tables)

This paper contains 74 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Reasoning Prompting
  4. Multi-Agent Reasoning
  5. $\textsc{OptAgent}\xspace$ Framework
  6. Problem Definition
  7. Framework Overview
  8. Initial Graph Setup
  9. Agent Profiling and Force Decoding
  10. Connection Initialization
  11. Verbal Reinforcement Learning
  12. Reflection
  13. Action
  14. Training Process
  15. Inference Process
  16. ...and 59 more sections

Figures (5)

  • Figure 1: Overview of $\textsc{OptAgent}\xspace$ framework. The overall pipeline is on the left side; an example process for verbal reinforcement learning is shown on the right.
  • Figure 2: Results on Creative Writing, measured in terms of coherence scores.
  • Figure 3: Training Convergence Trend for OptAgent under the 5-Agent Setting.
  • Figure 4: Case Study on the agent interaction graph. Numbers beside the connections signify the order of the interactions made. The collaboration frameworks is trained on the GSM-PLUS dataset.
  • Figure 5: Case Study on the agent interaction graph. Numbers beside the connections signify the order of the interactions made. The collaboration frameworks is trained on the Creative Writing Task.