Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Evolutionary Optimization for Resource-Efficient Neural Network Training

Ilia Revin, Leon Strelkov, Vadim A. Potemkin, Ivan Kireev, Andrey Savchenko

TL;DR

PETRA addresses the need for resource-efficient neural network training by introducing a domain-general AutoML framework that uses evolutionary optimization to automatically compose training pipelines with pruning, quantization, and loss-based regularization. It formalizes a multi-objective Pareto optimization balancing accuracy, latency, throughput, and size, and demonstrates across financial, image, and time-series tasks that significant compression (up to 85% model size reduction) and latency/throughput gains are achievable with minimal degradation in target metrics. The approach leverages loss regularization and low-rank decomposition, integrates diverse PEFT techniques, and employs mutation-based search with early-stopping to robustly identify Pareto-optimal pipelines. The results support PETRA's potential as a practical, domain-agnostic AutoML tool for deploying compact, efficient neural networks in diverse deployment settings, while noting limitations in precision-sensitive tasks and search overhead that warrant future hardware-aware and meta-learning enhancements.

Abstract

There are many critical challenges in optimizing neural network models, including distributed computing, compression techniques, and efficient training, regardless of their application to specific tasks. Solving such problems is crucial because the need for scalable and resource-efficient models is increasing. To address these challenges, we have developed a new automated machine learning (AutoML) framework, Parameter Efficient Training with Robust Automation (PETRA). It applies evolutionary optimization to model architecture and training strategy. PETRA includes pruning, quantization, and loss regularization. Experimental studies on real-world data with financial event sequences, as well as image and time-series -- benchmarks, demonstrate PETRA's ability to improve neural model performance and scalability -- namely, a significant decrease in model size (up to 75%) and latency (up to 33%), and an increase in throughput (by 13%) without noticeable degradation in the target metric.

Automated Evolutionary Optimization for Resource-Efficient Neural Network Training

TL;DR

PETRA addresses the need for resource-efficient neural network training by introducing a domain-general AutoML framework that uses evolutionary optimization to automatically compose training pipelines with pruning, quantization, and loss-based regularization. It formalizes a multi-objective Pareto optimization balancing accuracy, latency, throughput, and size, and demonstrates across financial, image, and time-series tasks that significant compression (up to 85% model size reduction) and latency/throughput gains are achievable with minimal degradation in target metrics. The approach leverages loss regularization and low-rank decomposition, integrates diverse PEFT techniques, and employs mutation-based search with early-stopping to robustly identify Pareto-optimal pipelines. The results support PETRA's potential as a practical, domain-agnostic AutoML tool for deploying compact, efficient neural networks in diverse deployment settings, while noting limitations in precision-sensitive tasks and search overhead that warrant future hardware-aware and meta-learning enhancements.

Abstract

There are many critical challenges in optimizing neural network models, including distributed computing, compression techniques, and efficient training, regardless of their application to specific tasks. Solving such problems is crucial because the need for scalable and resource-efficient models is increasing. To address these challenges, we have developed a new automated machine learning (AutoML) framework, Parameter Efficient Training with Robust Automation (PETRA). It applies evolutionary optimization to model architecture and training strategy. PETRA includes pruning, quantization, and loss regularization. Experimental studies on real-world data with financial event sequences, as well as image and time-series -- benchmarks, demonstrate PETRA's ability to improve neural model performance and scalability -- namely, a significant decrease in model size (up to 75%) and latency (up to 33%), and an increase in throughput (by 13%) without noticeable degradation in the target metric.

Paper Structure

This paper contains 13 sections, 6 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. AutoML frameworks
  4. Parameter-efficient training methods
  5. Proposed approach
  6. Loss Regularization and Low-Rank Decomposition
  7. Pruning and Quantization
  8. PETRA framework
  9. Experimental study
  10. Datasets
  11. Equipment
  12. Results and Discussion
  13. Conclusions

Figures (4)

  • Figure 1: The proposed PETRA framework
  • Figure 2: Percentage Change in Metrics by Pipeline for Model ResNet and CIFAR10 dataset
  • Figure 3: Percentage Change in Metrics by Pipeline for Model ResNet and ImageNette dataset
  • Figure 4: Percentage Change in Metrics by Pipeline for Model InceptionTime and ApplianceEnergy dataset