Balancing Fairness and Performance in Multi-User Spark Workloads with Dynamic Scheduling (extended version)

TL;DR

This work tackles the challenge of balancing fairness across multiple users and minimizing job response times in multi-user Spark workloads. It introduces User Weighted Fair Queuing (UWFQ), a scheduler that uses 2-level virtual time (user and global) to enforce bounded user-job fairness while prioritizing jobs with earlier virtual finish times, and it augments this with dynamic runtime partitioning to mitigate task skew and priority inversions. The approach is implemented inside Spark, with integration at scheduling and partitioning points and a grace-period mechanism to handle runtime estimation drift. Empirical evaluation on micro- and macro-benchmarks, including Google traces, shows that UWFQ reduces average response times for small jobs by up to 74% and generally outperforms Spark’s built-in fair scheduler and CFQ, especially when combined with runtime partitioning. The results support the practicality of UWFQ for industrial analytics environments that require both fairness and low latency across diverse users and workloads.

Abstract

Apache Spark is a widely adopted framework for large-scale data processing. However, in industrial analytics environments, Spark's built-in schedulers, such as FIFO and fair scheduling, struggle to maintain both user-level fairness and low mean response time, particularly in long-running shared applications. Existing solutions typically focus on job-level fairness which unintentionally favors users who submit more jobs. Although Spark offers a built-in fair scheduler, it lacks adaptability to dynamic user workloads and may degrade overall job performance. We present the User Weighted Fair Queuing (UWFQ) scheduler, designed to minimize job response times while ensuring equitable resource distribution across users and their respective jobs. UWFQ simulates a virtual fair queuing system and schedules jobs based on their estimated finish times under a bounded fairness model. To further address task skew and reduce priority inversions, which are common in Spark workloads, we introduce runtime partitioning, a method that dynamically refines task granularity based on expected runtime. We implement UWFQ within the Spark framework and evaluate its performance using multi-user synthetic workloads and Google cluster traces. We show that UWFQ reduces the average response time of small jobs by up to 74% compared to existing built-in Spark schedulers and to state-of-the-art fair scheduling algorithms.

Figures (7)

• Figure 1: Execution of user-constructed RDDs on cluster resources.
• Figure 2: System outline. Here, user 1 schedules a long-running job (blue), followed by a short job (red) from user 2 and another smaller job by user 1 (green). All users and jobs benefit from a low response time thanks to dynamic partitioning and fair scheduling.
• Figure 3: Impact of task skew on job runtime. The default task partitioning leads to high completion time, in (\ref{['subfig: skew bad']}), while UWFQ's dynamic partitioning, in (\ref{['subfig: skew good']}), reduces it. The green arrows indicate the finish time of the job.
• Figure 4: Priority inversion of red (high priority) and blue (low priority) jobs. In (\ref{['subfig: inversion bad']}) the red job has a smaller runtime and higher priority, but only executes once the blue job's tasks have finished. But in (\ref{['subfig: inversion good']}), thanks to UWFQ's dynamic partitioning, the red job is able to execute earlier. The red arrow indicates the expected finish time of the red job.
• Figure 5: Empirical CDFs for infrequent users in scenario 1.

Theorems & Definitions (5)

• lemma 1
• lemma 2
• theorem 1
• theorem 2
• corollary 1