MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era

TL;DR

MegaFlow tackles the infrastructure bottlenecks of large-scale agent training by introducing a three-service architecture that decouples model computation, agent coordination, and environment provisioning. Through Model Service, Agent Service, and Environment Service, the system enables elastic, event-driven orchestration and scalable resource management, validated on tens of thousands of concurrent SWE tasks. The evaluation shows MegaFlow achieves a $32\%$ cost reduction and scales consistently to up to $10{,}000$ concurrent tasks, outperforming centralized baselines and demonstrating robust resource utilization and favorable latency characteristics. The approach provides a production-ready foundation with broad agent-framework compatibility, offering significant practical impact for research and deployment of agentic AI in complex, real-world environments.

Abstract

The rapid development of interactive and autonomous AI systems signals our entry into the agentic era. Training and evaluating agents on complex agentic tasks such as software engineering and computer use requires not only efficient model computation but also sophisticated infrastructure capable of coordinating vast agent-environment interactions. However, no open-source infrastructure can effectively support large-scale training and evaluation on such complex agentic tasks. To address this challenge, we present MegaFlow, a large-scale distributed orchestration system that enables efficient scheduling, resource allocation, and fine-grained task management for agent-environment workloads. MegaFlow abstracts agent training infrastructure into three independent services (Model Service, Agent Service, and Environment Service) that interact through unified interfaces, enabling independent scaling and flexible resource allocation across diverse agent-environment configurations. In our agent training deployments, MegaFlow successfully orchestrates tens of thousands of concurrent agent tasks while maintaining high system stability and achieving efficient resource utilization. By enabling such large-scale agent training, MegaFlow addresses a critical infrastructure gap in the emerging agentic AI landscape.

Figures (6)

• Figure 1: The proposed three-service architecture for agent training. (left) The Environment Service provides diverse interactive execution environments and returns feedback (observations, rewards, termination signals) in response to actions. (middle) The Agent Service orchestrates interaction, collects trajectories, and manages experiences. (right) The Model Service supports both inference (returning policies from context) and training (updating from experiences).
• Figure 2: The architecture of MegaFlow. (bottom) The Model Service provides inference and training capabilities through various engines and distributed frameworks. (middle) The Agent Service coordinates execution strategies, integrates with agent frameworks, and manages experience feedback loops. (top) The Environment Service provides containerized execution environments and handles distributed task scheduling.
• Figure 3: Throughput scaling and cost comparison between MegaFlow and centralized approaches. (Left) Total execution time showing MegaFlow's consistent performance versus centralized degradation. (Right) Total cost comparison with 32% cost reduction at 2,000 concurrent tasks. Data represents bootstrap sampling from over 130,000 production records.
• Figure 4: Resource utilization patterns across normalized execution time. (Left) CPU utilization: centralized peak at 25% versus MegaFlow's stable 5-10%. (Right) Memory utilization: centralized peak at 50% versus MegaFlow's consistent 12%. Shaded areas represent 95% confidence intervals.
• Figure 5: End-to-end latency breakdown and environment startup scaling comparison. (Left) Total execution times: MegaFlow Persistent (75 min), Ephemeral (90 min), and High-Spec Centralized (110 min). (Right) Environment startup time scaling showing centralized degradation (1-13 min) versus MegaFlow's stable performance.

Theorems & Definitions (2)

• definition 1
• definition 2