Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework

Yiyang Zhao, Yunzhuo Liu, Bo Jiang, Tian Guo

TL;DR

CE-NAS addresses the carbon footprint of neural architecture search by introducing a carbon-aware end-to-end framework that dynamically allocates GPU resources between energy-efficient one/few-shot NAS sampling and energy-intensive vanilla NAS evaluation. It combines a time-series Transformer to forecast carbon intensity, an RL-based GPU allocation policy, and the LaMOO multi-objective optimizer to partition the search space and drive Pareto-efficient architecture discovery. Empirical results on HW-NASBench, NasBench301, CIFAR-10, and ImageNet show substantial carbon reductions (up to multiple-fold) with competitive or state-of-the-art accuracy and latency, demonstrating practical gains for carbon-aware NAS workflows. The framework is open to time-budget adaptations and demonstrates that forecast-informed resource management can match or exceed traditional NAS performance while dramatically lowering carbon emissions.

Abstract

This work presents a novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. The proposed framework CE-NAS addresses the key challenge of high carbon cost associated with NAS by exploring the carbon emission variations of energy and energy differences of different NAS algorithms. At the high level, CE-NAS leverages a reinforcement-learning agent to dynamically adjust GPU resources based on carbon intensity, predicted by a time-series transformer, to balance energy-efficient sampling and energy-intensive evaluation tasks. Furthermore, CE-NAS leverages a recently proposed multi-objective optimizer to effectively reduce the NAS search space. We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS datasets and open-domain NAS tasks. For example, on the HW-NasBench dataset, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For open-domain NAS tasks, CE-NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and a carbon consumption of 38.53 lbs of CO2. On ImageNet, our searched model achieves 80.6% top-1 accuracy with a 0.78 ms TensorRT latency using FP16 on NVIDIA V100, consuming only 909.86 lbs of CO2, making it comparable to other one-shot-based NAS baselines.

CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework

TL;DR

CE-NAS addresses the carbon footprint of neural architecture search by introducing a carbon-aware end-to-end framework that dynamically allocates GPU resources between energy-efficient one/few-shot NAS sampling and energy-intensive vanilla NAS evaluation. It combines a time-series Transformer to forecast carbon intensity, an RL-based GPU allocation policy, and the LaMOO multi-objective optimizer to partition the search space and drive Pareto-efficient architecture discovery. Empirical results on HW-NASBench, NasBench301, CIFAR-10, and ImageNet show substantial carbon reductions (up to multiple-fold) with competitive or state-of-the-art accuracy and latency, demonstrating practical gains for carbon-aware NAS workflows. The framework is open to time-budget adaptations and demonstrates that forecast-informed resource management can match or exceed traditional NAS performance while dramatically lowering carbon emissions.

Abstract

This work presents a novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. The proposed framework CE-NAS addresses the key challenge of high carbon cost associated with NAS by exploring the carbon emission variations of energy and energy differences of different NAS algorithms. At the high level, CE-NAS leverages a reinforcement-learning agent to dynamically adjust GPU resources based on carbon intensity, predicted by a time-series transformer, to balance energy-efficient sampling and energy-intensive evaluation tasks. Furthermore, CE-NAS leverages a recently proposed multi-objective optimizer to effectively reduce the NAS search space. We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS datasets and open-domain NAS tasks. For example, on the HW-NasBench dataset, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For open-domain NAS tasks, CE-NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and a carbon consumption of 38.53 lbs of CO2. On ImageNet, our searched model achieves 80.6% top-1 accuracy with a 0.78 ms TensorRT latency using FP16 on NVIDIA V100, consuming only 909.86 lbs of CO2, making it comparable to other one-shot-based NAS baselines.
Paper Structure (34 sections, 4 equations, 8 figures, 6 tables)

This paper contains 34 sections, 4 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Design of a Carbon-Efficient NAS Framework
  4. CE-NAS Overview
  5. Search Initialization
  6. Energy-Efficient Architecture Sampling
  7. Multi-objective search space partition.
  8. Energy-Consuming Architecture Evaluation
  9. Carbon-Efficient GPU Allocation
  10. Design of RL-Based GPU Allocation Strategy
  11. State Input.
  12. Network architecture.
  13. Action.
  14. Rewards.
  15. Policy gradient.
  16. ...and 19 more sections

Figures (8)

  • Figure 1: An overview of CE-NAS. The sampling and evaluation tasks will be dispatched with different GPU resources based on carbon emission intensity during the neural architecture search.
  • Figure 2: Transformer-based carbon intensity predictor. Our predictor makes forecasts that perfectly match actual values over three different regions.
  • Figure 3: Carbon traces from electricityMap. This carbon trace is based on the US-CAL-CISO data from 2021, specifically covering the period from 0:00, July 2, 2021, to 8:00, July 4, 2021. The blue trace is its actual carbon trace and the yellow trace is the prediction trace by our carbon predictor described in sec. \ref{['sec:carbon_forecast']}
  • Figure 4: Search progress over time. CE-NAS has the lowest relative carbon emission while achieving the second best $HV_{\mathrm{log\_diff}}$ on HW-NAS-Bench, and CE-NAS has the lowest relative carbon emission while achieving the second best $HV$ on NasBench301.
  • Figure 5: Search efficiency under carbon emission constraints. These results are based on NasBench201 with carbon trace showing in fig. \ref{['fig:trace']}, and we ran each method ten times.
  • ...and 3 more figures