Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alternative Mixed Integer Linear Programming Optimization for Joint Job Scheduling and Data Allocation in Grid Computing

Shengyu Feng, Jaehyung Kim, Yiming Yang, Joseph Boudreau, Tasnuva Chowdhury, Adolfy Hoisie, Raees Khan, Ozgur O. Kilic, Scott Klasky, Tatiana Korchuganova, Paul Nilsson, Verena Ingrid Martinez Outschoorn, David K. Park, Norbert Podhorszki, Yihui Ren, Frederic Suter, Sairam Sri Vatsavai, Wei Yang, Shinjae Yoo, Tadashi Maeno, Alexei Klimentov

TL;DR

The paper tackles the problem of jointly optimizing job scheduling and data allocation in grid computing to minimize makespan. It introduces AlterMILP, an alternating MILP approach that decomposes nonlinear quadratic constraints into tractable MILP subproblems by selectively fixing variable subsets, and solves them with off-the-shelf MILP solvers. Empirical results show AlterMILP consistently outperforms heuristic and MIQP-based baselines across multiple grid scales, with robustness to hyper-parameter settings. The method enables scalable, high-quality coordination of heterogeneous grid resources, offering practical impact for data-intensive HPC workloads.

Abstract

This paper presents a novel approach to the joint optimization of job scheduling and data allocation in grid computing environments. We formulate this joint optimization problem as a mixed integer quadratically constrained program. To tackle the nonlinearity in the constraint, we alternatively fix a subset of decision variables and optimize the remaining ones via Mixed Integer Linear Programming (MILP). We solve the MILP problem at each iteration via an off-the-shelf MILP solver. Our experimental results show that our method significantly outperforms existing heuristic methods, employing either independent optimization or joint optimization strategies. We have also verified the generalization ability of our method over grid environments with various sizes and its high robustness to the algorithm hyper-parameters.

Alternative Mixed Integer Linear Programming Optimization for Joint Job Scheduling and Data Allocation in Grid Computing

TL;DR

The paper tackles the problem of jointly optimizing job scheduling and data allocation in grid computing to minimize makespan. It introduces AlterMILP, an alternating MILP approach that decomposes nonlinear quadratic constraints into tractable MILP subproblems by selectively fixing variable subsets, and solves them with off-the-shelf MILP solvers. Empirical results show AlterMILP consistently outperforms heuristic and MIQP-based baselines across multiple grid scales, with robustness to hyper-parameter settings. The method enables scalable, high-quality coordination of heterogeneous grid resources, offering practical impact for data-intensive HPC workloads.

Abstract

This paper presents a novel approach to the joint optimization of job scheduling and data allocation in grid computing environments. We formulate this joint optimization problem as a mixed integer quadratically constrained program. To tackle the nonlinearity in the constraint, we alternatively fix a subset of decision variables and optimize the remaining ones via Mixed Integer Linear Programming (MILP). We solve the MILP problem at each iteration via an off-the-shelf MILP solver. Our experimental results show that our method significantly outperforms existing heuristic methods, employing either independent optimization or joint optimization strategies. We have also verified the generalization ability of our method over grid environments with various sizes and its high robustness to the algorithm hyper-parameters.

Paper Structure

This paper contains 22 sections, 13 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Framework Overview
  4. Data objects.
  5. Jobs.
  6. Computational nodes.
  7. Storage nodes.
  8. Problem Formulation
  9. Job scheduling.
  10. Data allocation.
  11. Joint Job Scheduling and Data Allocation via Alternative MILP Optimization
  12. MILP Problem Decomposition by Selectively Fixing Decision Variables
  13. AlterMILP: Efficient Joint Optimization by Alternative MILP
  14. Experiments
  15. Experimental Setups
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Overview of the grid computing environment. It consists of the jobs, computational nodes (CNs), storage nodes (SNs) and data objects. The remote SNs are connected with local SNs through a low-speed Wide Area Network (WAN) and CNs are connected with local SNs via a high-speed Local Area Network (LAN).
  • Figure 2: Illustration of the makespan. The running time of each job consists of both the data downloading time and execution time. The completion time of the last job marks the makespan of the job batch.
  • Figure 3: Analyses of optimization-based methods (MinTrans, MinExe, AlterMILP) by varying time budget for the optimization. Since MinTrans largely underperforms other methods, we exclude it in Grid-medium/large for better visualization.
  • Figure 4: Analyses of AlterMILP by varying the number of iterations $T$ on Grid-medium. (Left) Divided Time per Iter.: total time budget $B$ is fixed, but $T$ is increased (i.e., less time per iteration). (Right) Same Time per Iter.: using the same time per iteration, i.e., total time budget $B$ is enlarged with increased $T$.