Can AI autonomously build, operate, and use the entire data stack?

TL;DR

The paper proposes Agentic DataOps to achieve autonomous management of the entire data stack, addressing enterprise data management's complexity. It analyzes how intelligent agents and LLM-based orchestration could design infrastructure, discover data, create flows, govern data, and generate insights across the lifecycle, highlighting both opportunities and challenges. It outlines foundational capabilities—continuous learning, grounding, planning, governance, multi-agent collaboration, benchmarking, and community participation—and suggests incremental, cross-layer development with strong emphasis on observability and human oversight. The work aims to spark debate, guide research, and catalyze practical progress toward self-managing data estates.

Abstract

Enterprise data management is a monumental task. It spans data architecture and systems, integration, quality, governance, and continuous improvement. While AI assistants can help specific persona, such as data engineers and stewards, to navigate and configure the data stack, they fall far short of full automation. However, as AI becomes increasingly capable of tackling tasks that have previously resisted automation due to inherent complexities, we believe there is an imminent opportunity to target fully autonomous data estates. Currently, AI is used in different parts of the data stack, but in this paper, we argue for a paradigm shift from the use of AI in independent data component operations towards a more holistic and autonomous handling of the entire data lifecycle. Towards that end, we explore how each stage of the modern data stack can be autonomously managed by intelligent agents to build self-sufficient systems that can be used not only by human end-users, but also by AI itself. We begin by describing the mounting forces and opportunities that demand this paradigm shift, examine how agents can streamline the data lifecycle, and highlight open questions and areas where additional research is needed. We hope this work will inspire lively debate, stimulate further research, motivate collaborative approaches, and facilitate a more autonomous future for data systems.

Figures (2)

• Figure 1: Illustration of the Agentic DataOps approach, detailing its core processes for an autonomous data stack and an example application in fund performance forecasting using specialized agents. Left: A high-level summary of the main stages in a data stack starting with designing and building the storage infrastructure to generating insights. Right: Example of changes moving back and forth across the entire stack. For instance, based on poor analytical performance, more data engineering is applied on the underlying data. However, the resulting outcome is still not satisfactory, and consequently additional data sources are added, which in turn requires updating the storage infrastructure and subsequently triggers the data ingestion and processing steps, before a new analytical model is built.
• Figure 2: Architecture of an Agentic DataOps system showing hierarchical layers of the data stack and corresponding specialized agents. The figure illustrates that optimal control, fault tolerance, and continuous feedback propagation through observability are needed for an autonomously deployed system. Controlling such an autonomous stack is challenging due to the sheer scale and interdependence of tasks, diverse requirements for planning, and critical constraints such as efficiency, governance, factuality, and external steerability—whether by humans or other AI systems. Note that the shown list of agents is not complete and that an agent might not just call tools, but invoke additional agents as well.