Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performance and Stability of Barrier Mode Parallel Systems with Heterogeneous and Redundant Jobs

Brenton Walker, Markus Fidler

TL;DR

This work analyzes Barrier Execution Mode (BEM) in parallel systems, focusing on blocking start and departure barriers and the (s,k,l) redundant-task approach to mitigate stragglers. It derives stability bounds for 1- and 2-barrier configurations, extends the analysis to hybrid systems with a mix of barrier and non-barrier jobs, and uses max-plus and stochastic network calculus to bound waiting and sojourn times. The authors validate theoretical results with Spark-based experiments, uncovering substantial overhead from barrier scheduling and proposing an overhead model that matches observed behavior. The findings highlight conditions under which barrier and redundancy strategies improve stability and throughput, offering practical guidance for deploying barrier-enabled parallel workloads.

Abstract

In some models of parallel computation, jobs are split into smaller tasks and can be executed completely asynchronously. In other situations the parallel tasks have constraints that require them to synchronize their start and possibly departure times. This is true of many parallelized machine learning workloads, and the popular Apache Spark processing engine has recently added support for Barrier Execution Mode, which allows users to add such barriers to their jobs. These barriers necessarily result in idle periods on some of the workers, which reduces their stability and performance, compared to equivalent workloads with no barriers. In this paper we will consider and analyze the stability and performance penalties resulting from barriers. We include an analysis of the stability of $(s,k,l)$ barrier systems that allow jobs to depart after $l$ out of $k$ of their tasks complete. We also derive and evaluate performance bounds for hybrid barrier systems servicing a mix of jobs, both with and without barriers, and with varying degrees of parallelism. For the purely 1-barrier case we compare the bounds and simulation results to benchmark data from a standalone Spark system. We study the overhead in the real system, and based on its distribution we attribute it to the dual event and polling-driven mechanism used to schedule barrier-mode jobs. We develop a model for this type of overhead and validate it against the real system through simulation.

Performance and Stability of Barrier Mode Parallel Systems with Heterogeneous and Redundant Jobs

TL;DR

This work analyzes Barrier Execution Mode (BEM) in parallel systems, focusing on blocking start and departure barriers and the (s,k,l) redundant-task approach to mitigate stragglers. It derives stability bounds for 1- and 2-barrier configurations, extends the analysis to hybrid systems with a mix of barrier and non-barrier jobs, and uses max-plus and stochastic network calculus to bound waiting and sojourn times. The authors validate theoretical results with Spark-based experiments, uncovering substantial overhead from barrier scheduling and proposing an overhead model that matches observed behavior. The findings highlight conditions under which barrier and redundancy strategies improve stability and throughput, offering practical guidance for deploying barrier-enabled parallel workloads.

Abstract

In some models of parallel computation, jobs are split into smaller tasks and can be executed completely asynchronously. In other situations the parallel tasks have constraints that require them to synchronize their start and possibly departure times. This is true of many parallelized machine learning workloads, and the popular Apache Spark processing engine has recently added support for Barrier Execution Mode, which allows users to add such barriers to their jobs. These barriers necessarily result in idle periods on some of the workers, which reduces their stability and performance, compared to equivalent workloads with no barriers. In this paper we will consider and analyze the stability and performance penalties resulting from barriers. We include an analysis of the stability of barrier systems that allow jobs to depart after out of of their tasks complete. We also derive and evaluate performance bounds for hybrid barrier systems servicing a mix of jobs, both with and without barriers, and with varying degrees of parallelism. For the purely 1-barrier case we compare the bounds and simulation results to benchmark data from a standalone Spark system. We study the overhead in the real system, and based on its distribution we attribute it to the dual event and polling-driven mechanism used to schedule barrier-mode jobs. We develop a model for this type of overhead and validate it against the real system through simulation.

Paper Structure

This paper contains 19 sections, 3 theorems, 61 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Systems with Blocking Barriers
  3. Barriers in Apache Spark
  4. $(s,k,l)$ Barrier Systems
  5. 1 vs 2-Barrier Architectures
  6. Hybrid Barrier Systems
  7. Stability in Barrier Execution Mode
  8. Stability with Start and Departure Barriers
  9. Stability with Start Barriers
  10. Stability of 2-Barrier $(s,k,l)$ Systems
  11. Stability of 1-Barrier $(s,k,l)$ Systems
  12. Performance with Respect to Task Throughput
  13. Performance Bounds for BEM Systems
  14. Max-Plus System Model
  15. Performance Bounds for 1-Barrier BEM Systems
  16. ...and 4 more sections

Key Result

Lemma 1

Let $K(n)$ be a sequence of discrete random variables with support $[1,s]$. Consider a system with $s$ parallel work-conserving workers, servicing a mix of jobs with and without a blocking start barrier, where job $n$ comprises $K(n)$ tasks. Let $Q_i(n)$ denote the service time of task $i \in [1,K(n where Then the system is an $S^{G}(m,n)$ server.

Figures (9)

  • Figure 1: Single-queue asynchronous task scheduling vs 1 and 2-barrier BEM systems.
  • Figure 2: Maximum stable utilization for both 1 and 2-barrier BEM systems for varying numbers of workers. Lines are analytical results from Eqns. \ref{['eq:2barrier-stability']} and \ref{['eq:1barrier-stability-bound']}. Individual data points are from corresponding simulations forkulator.
  • Figure 3: Maximum stable utilization for 2-barrier $(s,k,l)$ BEM system with $s=k=16$ and varying $l$, from the bounds computed in section \ref{['sec:skl-2barrier-systems']}, and simulation. The left plot is for iid exponential task service times. The right plot is for a bimodal exponential service process, where there is a 10% probability of receiving slower service.
  • Figure 4: Maximum stable utilization for 1-barrier $(s,k,l)$ BEM system with $k=16$, $l=8$, and varying $s$, from the solution of the CTMC described in section \ref{['sec:skl-1barrier']}, and simulation.
  • Figure 5: The $10^{-6}$ quantile of service time, waiting time, and sojourn time for a system servicing a hybrid mix of BEM and non-BEM jobs, with utilization $\rho=0.7$. The $x$-axis spans the range from no BEM jobs to all BEM jobs.
  • ...and 4 more figures

Theorems & Definitions (7)

  • Definition 1: Max-plus server
  • Definition 2: $(\sigma,\rho)$-Arrival and Service Envelopes
  • Lemma 1: Generalized Barrier Execution Mode system
  • proof
  • Corollary 1
  • proof
  • Theorem 1: Barrier Execution Mode system