Autonomous Data Processing using Meta-Agents

TL;DR

ADP-MA tackles the rigidity of static data pipelines by introducing a hierarchical meta-agent framework that plans, orchestrates, and monitors end-to-end data processing. The system decomposes tasks into phases, dynamically instantiates ground-level agents with domain tools, and iteratively refines pipelines via a two-level critique and intelligent backtracking. It emphasizes context-aware optimization, progressive sampling, and sandboxed execution to enable scalable, robust data processing across diverse tasks. The approach is positioned against existing benchmarks and demonstrated through an interactive demo, with future evaluation planned on KRAMABENCH and related datasets.

Abstract

Traditional data processing pipelines are typically static and handcrafted for specific tasks, limiting their adaptability to evolving requirements. While general-purpose agents and coding assistants can generate code for well-understood data pipelines, they lack the ability to autonomously monitor, manage, and optimize an end-to-end pipeline once deployed. We present \textbf{Autonomous Data Processing using Meta-agents} (ADP-MA), a framework that dynamically constructs, executes, and iteratively refines data processing pipelines through hierarchical agent orchestration. At its core, \textit{meta-agents} analyze input data and task specifications to design a multi-phase plan, instantiate specialized \textit{ground-level agents}, and continuously evaluate pipeline performance. The architecture comprises three key components: a planning module for strategy generation, an orchestration layer for agent coordination and tool integration, and a monitoring loop for iterative evaluation and backtracking. Unlike conventional approaches, ADP-MA emphasizes context-aware optimization, adaptive workload partitioning, and progressive sampling for scalability. Additionally, the framework leverages a diverse set of external tools and can reuse previously designed agents, reducing redundancy and accelerating pipeline construction. We demonstrate ADP-MA through an interactive demo that showcases pipeline construction, execution monitoring, and adaptive refinement across representative data processing tasks.

Figures (9)

• Figure 1: Architecture for ADP-MA showing the three meta-agents (Orchestrator, Architect, Monitor), ground-level agent ecosystem, and supporting infrastructure including the workspace/sandbox and tool library.
• Figure 2: The Architect meta-agent workflow. The Architect receives task context and data summary from the Orchestrator, generates a high-level plan, refines it through a two-level critique loop (Level 1 for plan structure, Level 2 for phase expansion), and produces schema-contracted substep definitions that are dispatched to ground-level agents for execution.
• Figure 3: Live Mode initial state. The left panel provides a task input area and LLM selection dropdowns for the planning and coding models. The right panel shows the meta-agent pipeline progress bar and placeholders for stage details and generated code.
• Figure 4: Replay Mode showing the Data Understanding stage. The stage panel displays detected row count, column count, column types, and a natural-language summary of the dataset characteristics.
• Figure 5: Replay Mode showing the Plan stage. The detail panel displays the selected strategy (e.g., centralized) and the decomposed phases with their objectives, providing visibility into the Architect meta-agent's output.