Empowering Scientific Workflows with Federated Agents

TL;DR

The paper tackles the challenge of orchestrating autonomous, federated agentic workflows across diverse scientific infrastructure. It introduces Academy, a modular middleware that decouples agent behavior from execution and communication, enabling agents to operate across heterogeneous resources via a launcher and exchange, with a mailbox-based messaging layer and ProxyStore-based data optimizations. It contributes a Python-based agent model with Behavior, Agent, Handles, Exchange, and Launcher components, plus patterns for state checkpoints, migration, hierarchies, resource pools, and process-as-a-service, demonstrated through MOFA and decentralized learning case studies. The empirical evaluation across HPC platforms shows favorable startup and messaging performance, scalable deployment, and effective data-transfer optimizations, indicating that Academy can accelerate autonomous discovery by enabling robust, scalable agentic workflows in federated research environments.

Abstract

Agentic systems, in which diverse agents cooperate to tackle challenging problems, are exploding in popularity in the AI community. However, the agentic frameworks used to build these systems have not previously enabled use with research cyberinfrastructure. Here we introduce Academy, a modular and extensible middleware designed to deploy autonomous agents across the federated research ecosystem, including HPC systems, experimental facilities, and data repositories. To meet the demands of scientific computing, Academy supports asynchronous execution, heterogeneous resources, high-throughput data flows, and dynamic resource availability. It provides abstractions for expressing stateful agents, managing inter-agent coordination, and integrating computation with experimental control. We present microbenchmark results that demonstrate high performance and scalability in HPC environments. To demonstrate the breadth of applications that can be supported by agentic workflow designs, we also present case studies in materials discovery, decentralized learning, and information extraction in which agents are deployed across diverse HPC systems.

Figures (6)

• Figure 1: Cooperative agents, spanning federated research infrastructure (experimental facilities, computational systems, data storage), can enable agentic workflows that autonomously steer discovery.
• Figure 2: Agents and clients in Academy interact via handles to invoke actions asynchronously. Agents implement a behavior, defined by their actions, control loops, and state. Academy decouples the control and data planes through the launcher and exchange components that manage spawning agents and communication, respectively.
• Figure 3: (Top) Warm-start time for $n$ agents/actors between Academy (using the Parsl launcher), Dask Actors, and Ray Actors. Ray does not benefit from warm-starts because a new process is spawned for each actor. (Bottom) Time to execute 30 actions per agent/actor (weak scaling). Each action sleeps for 1 s. Note the Academy and Ray lines are overlapped.
• Figure 4: (Top) Time for a client to invoke a no-op action on an actor as a function of input and output payload size with different optimizations enabled on the distributed exchange. Two scenarios are considered: client and agent are at the same site (left) and different sites (right). (Bottom) Time for a client to invoke a chain of $n$ actions across $n$ agents with a payload size of 10 MB. Each action in the chain is a no-op that passes the input data along to the next agent, and returns the resulting data. The pass-by-reference optimization reduces communication costs among intermediate actions.
• Figure 5: Maximum no-op action throughput for a single agent requesting actions from $n$ worker agents. The handle multiplexing optimization improves performance by reducing the number of mailbox listener threads from $n$ to 1.