Trustworthy Scheduling for Big Data Applications

TL;DR

The paper tackles the challenge of resource sizing for containerized big data tasks under strict deadlines, where traditional schedulers act as black boxes and offer limited guidance. It introduces X-Sched, an explainable scheduling middleware that uses counterfactual explanations combined with Random Forests to produce minimal, actionable changes to container configurations (memory, CPU, replicas) that render a task schedulable within its deadline $t_k^{dl}$. Core contributions include the X-Sched Library with the XSchedTask abstraction, the X-Sched Runtime for generating actionable counterfactuals with an efficient pruning-based RF traversal, and an experimental evaluation on Alibaba traces demonstrating high feasibility and proximity of generated configurations. The work underscores the practical impact of offering transparent, interpretable scheduling decisions that help practitioners meet SLOs while reducing over- and under-provisioning in production containerized pipelines.

Abstract

Recent advances in modern containerized execution environments have resulted in substantial benefits in terms of elasticity and more efficient utilization of computing resources. Although existing schedulers strive to optimize performance metrics like task execution times and resource utilization, they provide limited transparency into their decision-making processes or the specific actions developers must take to meet Service Level Objectives (SLOs). In this work, we propose X-Sched, a middleware that uses explainability techniques to generate actionable guidance on resource configurations that makes task execution in containerized environments feasible, under resource and time constraints. X-Sched addresses this gap by integrating counterfactual explanations with advanced machine learning models, such as Random Forests, to efficiently identify optimal configurations. This approach not only ensures that tasks are executed in line with performance goals but also gives users clear, actionable insights into the rationale behind scheduling decisions. Our experimental results validated with data from real-world execution environments, illustrate the efficiency, benefits and practicality of our approach.

Figures (5)

• Figure 1: Task completion time (TCT) per CPU allocation
• Figure 2: Task completion time (TCT) per memory allocation
• Figure 3: X-Sched middleware
• Figure 4: Feasible Actions (the gray circle denotes the area of the feasible solutions)
• Figure 5: Density of Feasible Actions (blue star annotates the original instance)