Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Big Data Workload Profiling for Energy-Aware Cloud Resource Management

Milan Parikh, Aniket Abhishek Soni, Sneja Mitinbhai Shah, Ayush Raj Jha

TL;DR

The paper addresses rising energy consumption in data-center big data processing by proposing a workload-aware, predictive VM-placement framework that profiles CPU, memory, and I/O behavior from historical logs and real-time telemetry. It predicts energy-per-placement with $\hat{E}(W_i, h) = f_{\theta}(W_i, R_h)$ and ranks candidates to minimize total energy under SLA constraints, enabling adaptive consolidation. Experimental evaluation across Hadoop MapReduce, Spark MLlib, and ETL workloads shows energy savings of 15–20% with negligible SLA violations, validating the practicality of telemetry-driven workload profiling for green cloud operation. The approach is hardware-agnostic and compatible with DVFS and container orchestration, offering a scalable path to sustainable, multi-tenant cloud data centers.

Abstract

Cloud data centers face increasing pressure to reduce operational energy consumption as big data workloads continue to grow in scale and complexity. This paper presents a workload aware and energy efficient scheduling framework that profiles CPU utilization, memory demand, and storage IO behavior to guide virtual machine placement decisions. By combining historical execution logs with real time telemetry, the proposed system predicts the energy and performance impact of candidate placements and enables adaptive consolidation while preserving service level agreement compliance. The framework is evaluated using representative Hadoop MapReduce, Spark MLlib, and ETL workloads deployed on a multi node cloud testbed. Experimental results demonstrate consistent energy savings of 15 to 20 percent compared to a baseline scheduler, with negligible performance degradation. These findings highlight workload profiling as a practical and scalable strategy for improving the sustainability of cloud based big data processing environments.

Big Data Workload Profiling for Energy-Aware Cloud Resource Management

TL;DR

The paper addresses rising energy consumption in data-center big data processing by proposing a workload-aware, predictive VM-placement framework that profiles CPU, memory, and I/O behavior from historical logs and real-time telemetry. It predicts energy-per-placement with and ranks candidates to minimize total energy under SLA constraints, enabling adaptive consolidation. Experimental evaluation across Hadoop MapReduce, Spark MLlib, and ETL workloads shows energy savings of 15–20% with negligible SLA violations, validating the practicality of telemetry-driven workload profiling for green cloud operation. The approach is hardware-agnostic and compatible with DVFS and container orchestration, offering a scalable path to sustainable, multi-tenant cloud data centers.

Abstract

Cloud data centers face increasing pressure to reduce operational energy consumption as big data workloads continue to grow in scale and complexity. This paper presents a workload aware and energy efficient scheduling framework that profiles CPU utilization, memory demand, and storage IO behavior to guide virtual machine placement decisions. By combining historical execution logs with real time telemetry, the proposed system predicts the energy and performance impact of candidate placements and enables adaptive consolidation while preserving service level agreement compliance. The framework is evaluated using representative Hadoop MapReduce, Spark MLlib, and ETL workloads deployed on a multi node cloud testbed. Experimental results demonstrate consistent energy savings of 15 to 20 percent compared to a baseline scheduler, with negligible performance degradation. These findings highlight workload profiling as a practical and scalable strategy for improving the sustainability of cloud based big data processing environments.
Paper Structure (25 sections, 9 equations, 3 figures)

This paper contains 25 sections, 9 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Workload Profiling
  5. Prediction Engine
  6. Energy-Aware Scheduling and VM Placement
  7. Experimental Setup
  8. Testbed Infrastructure
  9. Workload Types
  10. Monitoring and Measurement
  11. Energy Instrumentation
  12. Baseline vs Optimized Comparison
  13. Results
  14. Energy Savings
  15. SLA Compliance and Performance
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Motivating context: energy use and cost factors in data centers that drive the need for energy-aware scheduling.
  • Figure 2: Overview of the predictive, workload-aware scheduling pipeline: profiling, classification, prediction, and energy-aware VM placement.
  • Figure 3: Observed energy reduction versus SLA compliance across evaluated workloads.