Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks

Gabriel Cortês, Nuno Lourenço, Paolo Romano, Penousal Machado

TL;DR

GreenFactory addresses the challenge of efficiently estimating neural-network performance during NAS by ensembling zero-cost proxies through a random-forest regressor to predict test accuracy directly. Using NATS-Bench, it constructs Green-NATS-Bench with 21 proxies plus size/complexity features, applies a 70-15-15 train/val/test split, and performs recursive feature elimination to balance accuracy and compute cost. Hyperparameter optimization yields two configurations, GreenFactory and GreenFactory-Fast, with strong, space- and dataset-robust correlations that surpass prior proxies. This ensemble approach reduces training requirements while delivering accurate performance estimates, enabling faster NAS with improved generalization. Future work will broaden benchmarks, incorporate more proxies, and explore richer dataset representations to enhance generalization.

Abstract

Determining the performance of a Deep Neural Network during Neural Architecture Search processes is essential for identifying optimal architectures and hyperparameters. Traditionally, this process requires training and evaluation of each network, which is time-consuming and resource-intensive. Zero-cost proxies estimate performance without training, serving as an alternative to traditional training. However, recent proxies often lack generalization across diverse scenarios and provide only relative rankings rather than predicted accuracies. To address these limitations, we propose GreenFactory, an ensemble of zero-cost proxies that leverages a random forest regressor to combine multiple predictors' strengths and directly predict model test accuracy. We evaluate GreenFactory on NATS-Bench, achieving robust results across multiple datasets. Specifically, GreenFactory achieves high Kendall correlations on NATS-Bench-SSS, indicating substantial agreement between its predicted scores and actual performance: 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120. Similarly, on NATS-Bench-TSS, we achieve correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120, showcasing its reliability in both search spaces.

GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks

TL;DR

GreenFactory addresses the challenge of efficiently estimating neural-network performance during NAS by ensembling zero-cost proxies through a random-forest regressor to predict test accuracy directly. Using NATS-Bench, it constructs Green-NATS-Bench with 21 proxies plus size/complexity features, applies a 70-15-15 train/val/test split, and performs recursive feature elimination to balance accuracy and compute cost. Hyperparameter optimization yields two configurations, GreenFactory and GreenFactory-Fast, with strong, space- and dataset-robust correlations that surpass prior proxies. This ensemble approach reduces training requirements while delivering accurate performance estimates, enabling faster NAS with improved generalization. Future work will broaden benchmarks, incorporate more proxies, and explore richer dataset representations to enhance generalization.

Abstract

Determining the performance of a Deep Neural Network during Neural Architecture Search processes is essential for identifying optimal architectures and hyperparameters. Traditionally, this process requires training and evaluation of each network, which is time-consuming and resource-intensive. Zero-cost proxies estimate performance without training, serving as an alternative to traditional training. However, recent proxies often lack generalization across diverse scenarios and provide only relative rankings rather than predicted accuracies. To address these limitations, we propose GreenFactory, an ensemble of zero-cost proxies that leverages a random forest regressor to combine multiple predictors' strengths and directly predict model test accuracy. We evaluate GreenFactory on NATS-Bench, achieving robust results across multiple datasets. Specifically, GreenFactory achieves high Kendall correlations on NATS-Bench-SSS, indicating substantial agreement between its predicted scores and actual performance: 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120. Similarly, on NATS-Bench-TSS, we achieve correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120, showcasing its reliability in both search spaces.
Paper Structure (18 sections, 5 figures, 9 tables)

This paper contains 18 sections, 5 figures, 9 tables.

Table of Contents

  1. Background and Related Work
  2. Zero-Cost Proxies
  3. NATS-Bench
  4. GreenFactory
  5. Data Collection
  6. Regression Model Selection
  7. Feature Selection
  8. Hyperparameter Tuning
  9. Evaluation
  10. Experimental Setup
  11. Results and Discussion
  12. Regression Model Selection
  13. Feature Selection
  14. Hyperparameter Tuning
  15. Evaluation
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Histograms of test accuracy for the NATS-Bench benchmark across the topology and size search spaces and the CIFAR-10, CIFAR-100, and ImageNet-16-120 datasets.
  • Figure 2: Relationship between the number of features and RMSE and mean computation time required to obtain the features for a single .
  • Figure 3: Feature importance on the two optimized models: GreenFactory and GreenFactory-Fast. For clarity, features with importance below 3% are grouped under the category Others in the GreenFactory model.
  • Figure 4: Features selected in the feature selection step.
  • Figure 5: Formulas of the selected GreenMachine zero-cost proxies.