Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic Meta-Orchestrator for Multi-task Copilots

Xiaofeng Zhu, Yunshen Zhou

TL;DR

The paper introduces the Agentic Meta-orchestrator (AMO), a framework to orchestrate multiple language-model-powered agents within Copilot-style ecosystems. AMO combines a top-k agent selection mechanism using a multi-level relevance learning-to-rank (with a Separator Class to handle unseen agents), memory-efficient LoRA arms for task-specific inference, and a meta-learning decision-tree planner to sequence agent usage deterministically. It demonstrates substantial performance gains on production-like copilots, notably the M365 E-Commerce Copilot and a Code Compliance Copilot, outperforming baselines such as AutoGen, Phi-3.5, and ChatGPT-4o in both agent orchestration and multi-task handling. The work also extends to hierarchical text classification with the uRank objective, showing scalability benefits when onboarding new agents and tasks. Overall, AMO offers a scalable, plug-in approach for multi-agent copilots with practical implications for enterprise AI systems across industries.

Abstract

Microsoft Copilot suites serve as the universal entry point for various agents skilled in handling important tasks, ranging from assisting a customer with product purchases to detecting vulnerabilities in corporate programming code. Each agent can be powered by language models, software engineering operations, such as database retrieval, and internal \& external knowledge. The repertoire of a copilot can expand dynamically with new agents. This requires a robust orchestrator that can distribute tasks from user prompts to the right agents. In this work, we propose an Agentic Meta-orchestrator (AMO) for handling multiple tasks and scalable agents in copilot services, which can provide both natural language and action responses. We will also demonstrate the planning that leverages meta-learning, i.e., a trained decision tree model for deciding the best inference strategy among various agents/models. We showcase the effectiveness of our AMO through two production use cases: Microsoft 365 (M365) E-Commerce Copilot and code compliance copilot. M365 E-Commerce Copilot advertises Microsoft products to external customers to promote sales success. The M365 E-Commerce Copilot provides up-to-date product information and connects to multiple agents, such as relational databases and human customer support. The code compliance copilot scans the internal DevOps code to detect known and new compliance issues in pull requests (PR).

Agentic Meta-Orchestrator for Multi-task Copilots

TL;DR

The paper introduces the Agentic Meta-orchestrator (AMO), a framework to orchestrate multiple language-model-powered agents within Copilot-style ecosystems. AMO combines a top-k agent selection mechanism using a multi-level relevance learning-to-rank (with a Separator Class to handle unseen agents), memory-efficient LoRA arms for task-specific inference, and a meta-learning decision-tree planner to sequence agent usage deterministically. It demonstrates substantial performance gains on production-like copilots, notably the M365 E-Commerce Copilot and a Code Compliance Copilot, outperforming baselines such as AutoGen, Phi-3.5, and ChatGPT-4o in both agent orchestration and multi-task handling. The work also extends to hierarchical text classification with the uRank objective, showing scalability benefits when onboarding new agents and tasks. Overall, AMO offers a scalable, plug-in approach for multi-agent copilots with practical implications for enterprise AI systems across industries.

Abstract

Microsoft Copilot suites serve as the universal entry point for various agents skilled in handling important tasks, ranging from assisting a customer with product purchases to detecting vulnerabilities in corporate programming code. Each agent can be powered by language models, software engineering operations, such as database retrieval, and internal \& external knowledge. The repertoire of a copilot can expand dynamically with new agents. This requires a robust orchestrator that can distribute tasks from user prompts to the right agents. In this work, we propose an Agentic Meta-orchestrator (AMO) for handling multiple tasks and scalable agents in copilot services, which can provide both natural language and action responses. We will also demonstrate the planning that leverages meta-learning, i.e., a trained decision tree model for deciding the best inference strategy among various agents/models. We showcase the effectiveness of our AMO through two production use cases: Microsoft 365 (M365) E-Commerce Copilot and code compliance copilot. M365 E-Commerce Copilot advertises Microsoft products to external customers to promote sales success. The M365 E-Commerce Copilot provides up-to-date product information and connects to multiple agents, such as relational databases and human customer support. The code compliance copilot scans the internal DevOps code to detect known and new compliance issues in pull requests (PR).
Paper Structure (16 sections, 1 equation, 9 figures, 2 tables)

This paper contains 16 sections, 1 equation, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Distribution and Semantic Learning-to-rank
  4. Multi-task Learning
  5. Agentic Workflow
  6. Proposed Agentic Meta-orchestrator
  7. Agentic Orchestrator of Top-k Selections
  8. LoRA Arms
  9. Meta Planning
  10. Production Use Case Experiments
  11. M365 E-Commerce Copilot
  12. Code Compliance Copilot
  13. Experimental Results
  14. Extension to Hierarchical Text Classification Results
  15. Limitations
  16. ...and 1 more sections

Figures (9)

  • Figure 1: Product Recommendation from ChatGPT
  • Figure 2: Agentic Orchestrator
  • Figure 3: LoRA Arms of Handling Multiple Tasks
  • Figure 4: A Meta-learning Decision Tree Model, in which Each Node is a Language Agent or Model
  • Figure 5: Classification F-1 with Growing Agents
  • ...and 4 more figures