Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents

Haoyuan Li, Mathias Funk, Aaqib Saeed

TL;DR

Helmsman tackles the manual bottleneck in federated learning system design by introducing a three-phase, multi-agent pipeline that converts high-level user specifications into deployable FL code. It combines interactive planning with self-reflection and human oversight, modular code generation across specialized agent teams, and autonomous sandboxed evaluation to iteratively refine solutions. The AgentFL-Bench benchmark of 16 tasks across five FL domains demonstrates that Helmsman-produced solutions can rival or surpass hand-crafted baselines, validating the feasibility of automated FL engineering. The work offers a practical path toward scalable, robust, and production-ready decentralized AI systems, with potential for further self-improvement through meta-optimization.

Abstract

Federated Learning (FL) offers a powerful paradigm for training models on decentralized data, but its promise is often undermined by the immense complexity of designing and deploying robust systems. The need to select, combine, and tune strategies for multifaceted challenges like data heterogeneity and system constraints has become a critical bottleneck, resulting in brittle, bespoke solutions. To address this, we introduce Helmsman, a novel multi-agent system that automates the end-to-end synthesis of federated learning systems from high-level user specifications. It emulates a principled research and development workflow through three collaborative phases: (1) interactive human-in-the-loop planning to formulate a sound research plan, (2) modular code generation by supervised agent teams, and (3) a closed-loop of autonomous evaluation and refinement in a sandboxed simulation environment. To facilitate rigorous evaluation, we also introduce AgentFL-Bench, a new benchmark comprising 16 diverse tasks designed to assess the system-level generation capabilities of agentic systems in FL. Extensive experiments demonstrate that our approach generates solutions competitive with, and often superior to, established hand-crafted baselines. Our work represents a significant step towards the automated engineering of complex decentralized AI systems.

Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents

TL;DR

Helmsman tackles the manual bottleneck in federated learning system design by introducing a three-phase, multi-agent pipeline that converts high-level user specifications into deployable FL code. It combines interactive planning with self-reflection and human oversight, modular code generation across specialized agent teams, and autonomous sandboxed evaluation to iteratively refine solutions. The AgentFL-Bench benchmark of 16 tasks across five FL domains demonstrates that Helmsman-produced solutions can rival or surpass hand-crafted baselines, validating the feasibility of automated FL engineering. The work offers a practical path toward scalable, robust, and production-ready decentralized AI systems, with potential for further self-improvement through meta-optimization.

Abstract

Federated Learning (FL) offers a powerful paradigm for training models on decentralized data, but its promise is often undermined by the immense complexity of designing and deploying robust systems. The need to select, combine, and tune strategies for multifaceted challenges like data heterogeneity and system constraints has become a critical bottleneck, resulting in brittle, bespoke solutions. To address this, we introduce Helmsman, a novel multi-agent system that automates the end-to-end synthesis of federated learning systems from high-level user specifications. It emulates a principled research and development workflow through three collaborative phases: (1) interactive human-in-the-loop planning to formulate a sound research plan, (2) modular code generation by supervised agent teams, and (3) a closed-loop of autonomous evaluation and refinement in a sandboxed simulation environment. To facilitate rigorous evaluation, we also introduce AgentFL-Bench, a new benchmark comprising 16 diverse tasks designed to assess the system-level generation capabilities of agentic systems in FL. Extensive experiments demonstrate that our approach generates solutions competitive with, and often superior to, established hand-crafted baselines. Our work represents a significant step towards the automated engineering of complex decentralized AI systems.
Paper Structure (49 sections, 8 figures, 17 tables)

This paper contains 49 sections, 8 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Motivation
  5. Agentic FL System
  6. Interactive and Verifiable Planning
  7. Agentic Plan Generation and Self-Reflection.
  8. Human-in-the-Loop (HITL) Verification.
  9. Modular Code Generation via Supervised Agent Teams
  10. Blueprint Decomposition.
  11. Collaborative Implementation.
  12. Autonomous Evaluation and Refinement
  13. Sandboxed Federated Simulation.
  14. Hierarchical Diagnosis.
  15. Automated Code Correction.
  16. ...and 34 more sections

Figures (8)

  • Figure 1: The automated FL development workflow of Helmsman. (a) Planning: A user query is refined into an actionable research plan via human-in-the-loop dialogue. (b) Coding: Specialized agent teams, managed by a Supervisor, collaboratively build a modular codebase. (c) Evaluation: The final code is autonomously tested and refined in a closed simulation loop until correct.
  • Figure 2: The intractable design space of FL, created by the combinatorial task of matching diverse challenges with specialized strategies.
  • Figure 3: Distribution of the 16 tasks in our AgentFL-Bench benchmark across five key FL research domains.
  • Figure 4: Code generation stability comparison on Q1 task across 3 independent runs. Claude Code (Claude-Sonnet-4.5) produces distinct folder structures and implementations in each run (a-c), demonstrating inconsistent code generation. In contrast, Helmsman maintains identical system structure across all runs (d), ensuring reproducibility and enabling plug-and-play modularity.
  • Figure 5: The complete interaction of Helmsman-Human during planning workflow on research query task 1. The input query type follows the original schema in AgentFL-Bench.
  • ...and 3 more figures