Barbarians at the Gate: How AI is Upending Systems Research

TL;DR

This paper argues that AI-driven approaches (ADRS) can transform systems research by automating solution discovery and evaluation, leveraging simulators for rapid verification. It presents an ADRS architecture with five components and demonstrates substantial performance gains over human-designed baselines across various domains, including MoE inference and cloud scheduling, often at low cost. The work highlights practical best practices, potential shifts in researcher roles toward problem formulation and strategy, and the need to adapt existing systems research practices for AI-assisted discovery. The results indicate ADRS can achieve near or above state-of-the-art performance within hours, underscoring its disruptive potential and the urgency of methodological adaptation in the systems community.

Abstract

Artificial Intelligence (AI) is starting to transform the research process as we know it by automating the discovery of new solutions. Given a task, the typical AI-driven approach is (i) to generate a set of diverse solutions, and then (ii) to verify these solutions and select one that solves the problem. Crucially, this approach assumes the existence of a reliable verifier, i.e., one that can accurately determine whether a solution solves the given problem. We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. We term this approach as AI-Driven Research for Systems (ADRS), which iteratively generates, evaluates, and refines solutions. Using penEvolve, an existing open-source ADRS instance, we present case studies across diverse domains, including load balancing for multi-region cloud scheduling, Mixture-of-Experts inference, LLM-based SQL queries, and transaction scheduling. In multiple instances, ADRS discovers algorithms that outperform state-of-the-art human designs (e.g., achieving up to 5.0x runtime improvements or 50% cost reductions). We distill best practices for guiding algorithm evolution, from prompt design to evaluator construction, for existing frameworks. We then discuss the broader implications for the systems community: as AI assumes a central role in algorithm design, we argue that human researchers will increasingly focus on problem formulation and strategic guidance. Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI.

Figures (4)

• Figure 1: The five stages of the systems research process. In this paper, we show how AI can automate the Solution and Evaluation stages (grey area).
• Figure 2: Time spent in various stages of the systems research process in systems, based on a survey of 31 PhD students. Algorithm Design (21.5%) and Evaluation (20.1%) together account for over 40% of total effort, highlighting a significant opportunity for leveraging AI to accelerate this process.
• Figure 3: The AI-driven Research for Systems (ADRS) architecture shown in the context of the systems research process (see in Figure \ref{['fig:research-process']}). ADRS (grey area) automates the Solution and Evaluation stages.
• Figure 4: Side-by-side comparison of the initial Uniform Progress policy and the evolved adaptive strategy. Key innovations in the evolved policy are highlighted.