Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spend More to Save More (SM2): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization

Daniel Geissler, Bo Zhou, Sungho Suh, Paul Lukowicz

TL;DR

SM$^2$ tackles the gap between performance-centric hyperparameter optimization and energy efficiency by integrating energy-aware metrics into a sequential Successive Halving framework. It uses exploratory pretraining and hardware power monitoring, notably $E_k$ computed from GPU power and time data, to prune inefficient hyperparameter configurations while preserving final model quality. The core contribution is the objective function $f(\alpha,\beta)$ that blends Performance $P$, Energy $E$, and Learning Rate $LR$, guiding batch-size and learning-rate decisions under energy constraints. Across ResNet-18/CIFAR-10, LSTM/Energy-Household, and Transformer/WikiText2 on multiple GPUs, SM$^2$ achieves substantial energy reductions with little to no degradation in performance, supporting a Spend More to Save More paradigm. The work lays a foundation for energy-aware machine learning workflows and points to future extensions for broader hardware support and multi-GPU parallelism.

Abstract

A fundamental step in the development of machine learning models commonly involves the tuning of hyperparameters, often leading to multiple model training runs to work out the best-performing configuration. As machine learning tasks and models grow in complexity, there is an escalating need for solutions that not only improve performance but also address sustainability concerns. Existing strategies predominantly focus on maximizing the performance of the model without considering energy efficiency. To bridge this gap, in this paper, we introduce Spend More to Save More (SM2), an energy-aware hyperparameter optimization implementation based on the widely adopted successive halving algorithm. Unlike conventional approaches including energy-intensive testing of individual hyperparameter configurations, SM2 employs exploratory pretraining to identify inefficient configurations with minimal energy expenditure. Incorporating hardware characteristics and real-time energy consumption tracking, SM2 identifies an optimal configuration that not only maximizes the performance of the model but also enables energy-efficient training. Experimental validations across various datasets, models, and hardware setups confirm the efficacy of SM2 to prevent the waste of energy during the training of hyperparameter configurations.

Spend More to Save More (SM2): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization

TL;DR

SM tackles the gap between performance-centric hyperparameter optimization and energy efficiency by integrating energy-aware metrics into a sequential Successive Halving framework. It uses exploratory pretraining and hardware power monitoring, notably computed from GPU power and time data, to prune inefficient hyperparameter configurations while preserving final model quality. The core contribution is the objective function that blends Performance , Energy , and Learning Rate , guiding batch-size and learning-rate decisions under energy constraints. Across ResNet-18/CIFAR-10, LSTM/Energy-Household, and Transformer/WikiText2 on multiple GPUs, SM achieves substantial energy reductions with little to no degradation in performance, supporting a Spend More to Save More paradigm. The work lays a foundation for energy-aware machine learning workflows and points to future extensions for broader hardware support and multi-GPU parallelism.

Abstract

A fundamental step in the development of machine learning models commonly involves the tuning of hyperparameters, often leading to multiple model training runs to work out the best-performing configuration. As machine learning tasks and models grow in complexity, there is an escalating need for solutions that not only improve performance but also address sustainability concerns. Existing strategies predominantly focus on maximizing the performance of the model without considering energy efficiency. To bridge this gap, in this paper, we introduce Spend More to Save More (SM2), an energy-aware hyperparameter optimization implementation based on the widely adopted successive halving algorithm. Unlike conventional approaches including energy-intensive testing of individual hyperparameter configurations, SM2 employs exploratory pretraining to identify inefficient configurations with minimal energy expenditure. Incorporating hardware characteristics and real-time energy consumption tracking, SM2 identifies an optimal configuration that not only maximizes the performance of the model but also enables energy-efficient training. Experimental validations across various datasets, models, and hardware setups confirm the efficacy of SM2 to prevent the waste of energy during the training of hyperparameter configurations.

Paper Structure

This paper contains 15 sections, 2 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Hyperparameter Optimization
  4. Energy Consumption Tracking
  5. Energy-Aware Training
  6. SM$^2$: Spend More to Save More
  7. Batch Size Optimization
  8. Learning Rate Optimization
  9. Objective Function
  10. Algorithm
  11. Experiments
  12. Setup
  13. SM$^2$ Evaluation
  14. Energy Compensation
  15. Conclusion and Future Work

Figures (2)

  • Figure 1: Computational analysis of loss curvature through the sliding window to select the area with the largest stable learning rate.
  • Figure 2: Evaluation of SM$^2$: Each row represents a different model and dataset combination; Columns represent Performance, Energy, and Learning Rate; Vertical Lines in the graph highlight exploratory epochs; Experiments were conducted on Nvidia RTX A6000.