Eudoxia: a FaaS scheduling simulator for the composable lakehouse

TL;DR

Eudoxia addresses the challenge of scheduling function-based workloads in a composable lakehouse by providing a deterministic, modular simulator. The three-component loop (Workload Generator, Scheduler, Executor) enables plug-and-play evaluation of custom scheduling policies against configurable workloads and resource models. The paper formalizes the scheduling problem in this setting, describes the simulator architecture, and demonstrates preliminary validation against real OLAP workloads (TPC-H) on Bauplan, showing realistic timing error and enabling cheap, repeatable policy exploration. The tool is released as open-source to support broader adoption and experimentation in the data-lakehouse community.

Abstract

Due to the variety of its target use cases and the large API surface area to cover, a data lakehouse (DLH) is a natural candidate for a composable data system. Bauplan is a composable DLH built on "spare data parts" and a unified Function-as-a-Service (FaaS) runtime for SQL queries and Python pipelines. While FaaS simplifies both building and using the system, it introduces novel challenges in scheduling and optimization of data workloads. In this work, starting from the programming model of the composable DLH, we characterize the underlying scheduling problem and motivate simulations as an effective tools to iterate on the DLH. We then introduce and release to the community Eudoxia, a deterministic simulator for scheduling data workloads as cloud functions. We show that Eudoxia can simulate a wide range of workloads and enables highly customizable user implementations of scheduling algorithms, providing a cheap mechanism for developers to evaluate different scheduling algorithms against their infrastructure.

Figures (3)

• Figure 1: Bauplan workers are off-the-shelf VMs, providing stateless compute capacity over object storage. Within an organization, users and machines (Apache Airflow, AWS Lambda on a schedule etc.) may submit interactive read-only queries (red) or asynchronous read-write pipelines (blue). What scheduling policy for functions can maximize a desired metric (e.g. throughput)?
• Figure 2: Simulator Architecture. Users set parameters and pass this to the initializer for Eudoxia which starts a loop of three components, the Workload Generator, Scheduler, and Executor. Once that loop completes, visualizers or other downstream applications can access execution statistics.
• Figure 3: Distribution of percent error of simulator estimates for runtime vs. real runtime for TPC-H queries at 10GB.