Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling

Jonathan Will, Nico Treide, Lauritz Thamsen, Odej Kao

TL;DR

The paper investigates resource efficiency in autoscaling for batch data processing by evaluating Spark autoscaling on Google Dataproc Serverless using a new bench dataset. It introduces benchspark, analyzes 234 Spark job executions across diverse configurations, and compares autoscaling against static baselines. Results show autoscaling provides limited additional resource efficiency over well-chosen static allocations, though it often improves runtimes; the main constraints arise from inelastic node sizes and fixed memory-to-CPU ratios. The work highlights the potential for cost optimization through per-job allocation strategies and cost-model adaption, guiding future research on more nuanced autoscaling policies for distributed dataflow systems.

Abstract

Distributed dataflow systems like Spark and Flink enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs is often challenging. For efficient execution, individual resource allocations, such as memory and CPU cores, must meet the specific resource requirements of the job. An alternative to selecting a static resource allocation for a job execution is autoscaling as implemented for example by Spark. In this paper, we evaluate the resource efficiency of autoscaling batch data processing jobs based on resource demand both conceptually and experimentally by analyzing a new dataset of Spark job executions on Google Dataproc Serverless. In our experimental evaluation, we show that there is no significant resource efficiency gain over static resource allocations. We found that the inherent conceptual limitations of such autoscaling approaches are the inelasticity of node size as well as the inelasticity of the ratio of memory to CPU cores.

Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling

TL;DR

The paper investigates resource efficiency in autoscaling for batch data processing by evaluating Spark autoscaling on Google Dataproc Serverless using a new bench dataset. It introduces benchspark, analyzes 234 Spark job executions across diverse configurations, and compares autoscaling against static baselines. Results show autoscaling provides limited additional resource efficiency over well-chosen static allocations, though it often improves runtimes; the main constraints arise from inelastic node sizes and fixed memory-to-CPU ratios. The work highlights the potential for cost optimization through per-job allocation strategies and cost-model adaption, guiding future research on more nuanced autoscaling policies for distributed dataflow systems.

Abstract

Distributed dataflow systems like Spark and Flink enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs is often challenging. For efficient execution, individual resource allocations, such as memory and CPU cores, must meet the specific resource requirements of the job. An alternative to selecting a static resource allocation for a job execution is autoscaling as implemented for example by Spark. In this paper, we evaluate the resource efficiency of autoscaling batch data processing jobs based on resource demand both conceptually and experimentally by analyzing a new dataset of Spark job executions on Google Dataproc Serverless. In our experimental evaluation, we show that there is no significant resource efficiency gain over static resource allocations. We found that the inherent conceptual limitations of such autoscaling approaches are the inelasticity of node size as well as the inelasticity of the ratio of memory to CPU cores.
Paper Structure (15 sections, 3 figures, 6 tables)

This paper contains 15 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Resource Allocation for Distributed Dataflow Jobs
  4. Autoscaling Resources for Distributed Dataflow Jobs
  5. Experimental Setup
  6. Spark Jobs
  7. Cloud Configurations
  8. Job Executions
  9. Experimental Evaluation
  10. Scaling Behavior of Dataproc Serverless
  11. Resource-Efficiency of Dataproc Serverless Autoscaling
  12. Impact of the Configuration Space Dimensions on Cost
  13. Impact of the Resource Cost Model
  14. Discussion
  15. Conclusion

Figures (3)

  • Figure 1: Time series of Dataproc Serverless executor allocation for executors in $[2,32]$.
  • Figure 2: Comparing the achieved runtime and the resulting resource use for each of the 18 jobs when using different executor allocation strategies. Normalization to 1 = lowest per job.
  • Figure 3: Comparing cloud configuration selection approaches for varying individual resource costs. Bold lines of the same color represent corresponding configurations in regular and serverless Dataproc.