PISA: An Adversarial Approach To Comparing Task Graph Scheduling Algorithms

TL;DR

This work introduces SAGA, a modular Python framework for benchmarking task-graph scheduling algorithms, and PISA, a simulated-annealing–based adversarial analysis method to uncover problem instances where a given algorithm underperforms relative to others. By benchmarking 15 algorithms across 16 datasets and then applying PISA, the authors demonstrate that algorithms with strong benchmarking performance can falter on adversarial instances, revealing practical performance boundaries. The results show pervasive, non-dominant relationships among schedulers and illustrate how adversarial instances persist even in application-specific, well-structured problem spaces. These insights offer a path toward more robust algorithm design and informed WFMS choices for heterogeneous compute networks.

Abstract

Scheduling a task graph representing an application over a heterogeneous network of computers is a fundamental problem in distributed computing. It is known to be not only NP-hard but also not polynomial-time approximable within a constant factor. As a result, many heuristic algorithms have been proposed over the past few decades. Yet it remains largely unclear how these algorithms compare to each other in terms of the quality of schedules they produce. We identify gaps in the traditional benchmarking approach to comparing task scheduling algorithms and propose a simulated annealing-based adversarial analysis approach called PISA to help address them. We also introduce SAGA, a new open-source library for comparing task scheduling algorithms. We use SAGA to benchmark 15 algorithms on 16 datasets and PISA to compare the algorithms in a pairwise manner. Algorithms that appear to perform similarly on benchmarking datasets are shown to perform very differently on adversarially chosen problem instances. Interestingly, the results indicate that this is true even when the adversarial search is constrained to selecting among well-structured, application-specific problem instances. This work represents an important step towards a more general understanding of the performance boundaries between task scheduling algorithms on different families of problem instances.

Figures (19)

• Figure 1: Example problem instance and schedule.
• Figure 2: Makespan Ratios of 15 algorithms evaluated on 16 datasets. Gradients depict performance on different problem instances in each dataset.
• Figure 3: Comparison of Scheduling Algorithms on Slightly Modified Networks
• Figure 4: Heatmap of makespan ratios for all 15 Algorithms compared to each other. The cell value (and color) for row $i$ and column $j$ indicates the makespan ratio for the worst-case instance found by PISA for scheduler $j$ against scheduler $i$.
• Figure 5: Problem Instance where HEFT performs $\approx 1.55$ times worse than CPoP.