NeurDB: On the Design and Implementation of an AI-powered Autonomous Database
Zhanhao Zhao, Shaofeng Cai, Haotian Gao, Hexiang Pan, Siqi Xiang, Naili Xing, Gang Chen, Beng Chin Ooi, Yanyan Shen, Yuncheng Wu, Meihui Zhang
TL;DR
NeurDB introduces an AI-powered autonomous DBMS that deeply embeds AI workflows in-database to support efficient AI analytics and fast-adaptive components that handle data and workload drift. It builds an in-database AI ecosystem with an AI Engine, data streaming, and a Model Manager that supports incremental updates, enabling PREDICT-based AI analytics directly in SQL. The system also deploys fast-adaptive learned concurrency control and a learned query optimizer to sustain performance under dynamic conditions, demonstrated by substantial gains in latency, throughput, and drift adaptability versus strong baselines. This unified, drift-aware architecture promises scalable, autonomous optimization and AI-enabled analytics for modern AI-centric applications.
Abstract
Databases are increasingly embracing AI to provide autonomous system optimization and intelligent in-database analytics, aiming to relieve end-user burdens across various industry sectors. Nonetheless, most existing approaches fail to account for the dynamic nature of databases, which renders them ineffective for real-world applications characterized by evolving data and workloads. This paper introduces NeurDB, an AI-powered autonomous database that deepens the fusion of AI and databases with adaptability to data and workload drift. NeurDB establishes a new in-database AI ecosystem that seamlessly integrates AI workflows within the database. This integration enables efficient and effective in-database AI analytics and fast-adaptive learned system components. Empirical evaluations demonstrate that NeurDB substantially outperforms existing solutions in managing AI analytics tasks, with the proposed learned components more effectively handling environmental dynamism than state-of-the-art approaches.