EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search

TL;DR

This work introduces an enhanced tabular benchmark encompassing data on energy consumption for varied architectures, and emphasizes the potential of EC-NAS by leveraging multi-objective optimization algorithms, revealing a balance between energy usage and accuracy.

Abstract

Energy consumption from the selection, training, and deployment of deep learning models has seen a significant uptick recently. This work aims to facilitate the design of energy-efficient deep learning models that require less computational resources and prioritize environmental sustainability by focusing on the energy consumption. Neural architecture search (NAS) benefits from tabular benchmarks, which evaluate NAS strategies cost-effectively through precomputed performance statistics. We advocate for including energy efficiency as an additional performance criterion in NAS. To this end, we introduce an enhanced tabular benchmark encompassing data on energy consumption for varied architectures. The benchmark, designated as EC-NAS, has been made available in an open-source format to advance research in energy-conscious NAS. EC-NAS incorporates a surrogate model to predict energy consumption, aiding in diminishing the energy expenditure of the dataset creation. Our findings emphasize the potential of EC-NAS by leveraging multi-objective optimization algorithms, revealing a balance between energy usage and accuracy. This suggests the feasibility of identifying energy-lean architectures with little or no compromise in performance.

Figures (6)

• Figure 1: Scatter plot of about 423k CNN architectures showing training energy ($E$) vs. validation performance ($P_v$) across four training budgets. Solutions in the top-right (red ellipse) prioritize performance at high energy costs. Joint optimization shifts preferred solutions to the left (green ellipse), indicating reduced energy with minimal performance loss.
• Figure 2: Scatter plot depicting the Kendall-Tau correlation coefficient between predicted and actual energy consumption (left) and the influence of training data size on test accuracy (right). Error bars are based on 10 random initializations.
• Figure 3: Aggregated impact of swapping one operator for another on energy consumption, training time, validation accuracy, and parameter count. The figure illustrates how changing a single operator can affect the different aspects of model performance, emphasizing the importance of selecting the appropriate operators to balance energy efficiency and performance.
• Figure 4: Energy consumption of models with DAGs where $|V| \leq 4$ on different GPUs. Models are organized by their average energy consumption for clarity.
• Figure 5: (Left) The attainment curve showing median solutions for 10 random initializations on the surrogate 7V space from EC-NAS dataset. (Center) A representation of the Pareto front for one run of SEMOA. (Right) Summary of metrics for the extrema and knee point architectures for one SEMOA run.