Learning to Orchestrate Agents in Natural Language with the Conductor

TL;DR

The paper introduces the RL Conductor, a 7B language model trained with reinforcement learning to autonomously design task decompositions, prompt-targeted subtasks, and communication topologies for a pool of worker LLMs. By learning end-to-end coordination strategies, the Conductor achieves state-of-the-art results on LiveCodeBench and GPQA, and demonstrates robustness to arbitrary agent pools and test-time recursion. Extensions such as randomized worker pools and recursive calling unlock new capabilities in adaptive compute budgeting and online scaling. The work provides a practical, scalable framework for language-model coordination and paves the way for future integrations of meta-agents across diverse domains.

Abstract

Powerful large language models (LLMs) from different providers have been expensively trained and finetuned to specialize across varying domains. In this work, we introduce a new kind of Conductor model trained with reinforcement learning to automatically discover powerful coordination strategies among LLMs. Our Conductor learns not only to design targeted communication topologies for effective agent-to-agent collaboration, but also to prompt engineer focused instructions to the LLMs to maximally leverage their individual capabilities. We show that, by learning optimal coordination strategies over pools of powerful worker LLMs, a 7B Conductor achieves significant performance gains beyond any individual worker, attaining state-of-the-art results in challenging reasoning benchmarks, such as LiveCodeBench and GPQA. By training with randomized agent pools, our conductor effectively adapts to arbitrary sets of open- and closed-source agents, meeting any user requirements. Furthermore, allowing the Conductor to select itself as a worker gives rise to recursive topologies, elevating performance with a new form of dynamic test-time scaling through online iterative adaptation. More broadly, ours is among the early work demonstrating language model coordination can be unlocked through RL, where powerful coordination strategies emerge naturally in LLMs through pure end-to-end reward maximization.

Figures (28)

• Figure 1: Our Conductor attains the state-of-the-art in GPQA and LiveCodeBench.
• Figure 2: The Conductor output. The Conductor responds with the entire coordination strategy.
• Figure 3: Emergence of powerful coordination strategies over training. Early in training, the Conductor issues sound subtasks, but does not tap useful collaborative strategies such as verification (bottom-right). Near convergence, the Conductor has learned to utilize planners, issue targeted instructions, instruct workers to share reasoning, and leverage verification and refinement (top-right), leading to the Conductor surpassing the worker agents’ performance on our training dataset (left).
• Figure 4: Conductor in-distribution evaluation against multi-agent methods and 5-turn reflection agent baselines. The Conductor surpasses all baselines by substantive margins, exemplifying the Conductor's ability to amplify the capabilities of its workers. Numerical results in Table \ref{['table:self_reflection_and_routing_comparison']}.
• Figure 5: Performance vs Efficiency. The Conductor far surpasses multi-agent baselines at a fraction of the cost. Scores are task-averages from Fig. \ref{['fig: indist']}. Numerical results in Table \ref{['table:self_reflection_and_routing_comparison']}