OMAC: A Broad Optimization Framework for LLM-Based Multi-Agent Collaboration

TL;DR

OMAC tackles the challenge of optimizing LLM-powered multi-agent systems across multi-step tasks by formalizing five optimization dimensions that cover both agent functionality and collaboration structure. It introduces a supervised optimization framework using a Semantic Initializer to generate diverse prompts and a Contrastive Comparator to reason about performance gaps, enabling both dimension-wise and iterative joint optimization. Across code generation, general reasoning, and arithmetic reasoning benchmarks, OMAC demonstrates significant performance gains over state-of-the-art baselines (including DyLAN) while addressing inference efficiency. The work advances systematic MAS design by providing a scalable, automated pathway to optimize both what agents do and how they collaborate.

Abstract

Agents powered by advanced large language models (LLMs) have demonstrated impressive capabilities across diverse complex applications. Recently, Multi-Agent Systems (MAS), wherein multiple agents collaborate and communicate with each other, have exhibited enhanced capabilities in complex tasks, such as high-quality code generation and arithmetic reasoning. However, the development of such systems often relies on handcrafted methods, and the literature on systematic design and optimization of LLM-based MAS remains limited. In this work, we introduce OMAC, a general framework designed for holistic optimization of LLM-based MAS. Specifically, we identify five key optimization dimensions for MAS, encompassing both agent functionality and collaboration structure. Building upon these dimensions, we first propose a general algorithm, utilizing two actors termed the Semantic Initializer and the Contrastive Comparator, to optimize any single dimension. Then, we present an algorithm for joint optimization across multiple dimensions. Extensive experiments demonstrate the superior performance of OMAC on code generation, arithmetic reasoning, and general reasoning tasks against state-of-the-art approaches.

Figures (7)

• Figure 1: OMAC optimization workflow for a single dimension.
• Figure 2: An example of optimization of a single dimension of OMAC.
• Figure 3: OMAC optimization framework for multiple dimensions.
• Figure 4: Performance during iterative optimization for multiple dimensions on arithmetic reasoning task. The X-axis represents the dimensions undergoing three iterations of optimization, while each point indicates the MAS's performance with the optimized dimensions on the test set. The error bar denotes the standard deviation.
• Figure 5: Performance during iterative optimization for multiple dimensions on code generation task. The X-axis represents the dimensions undergoing three iterations of optimization, while each point indicates the MAS's performance with the optimized dimensions on the test set. The error bar denotes the standard deviation.