Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Maintaining Performance with Less Data

Dominic Sanderson, Tatiana Kalgonova

TL;DR

The paper tackles the high cost and environmental impact of training image classifiers by introducing three dynamic data-use methods to reduce input data during training. It evaluates these methods—Data Step, Data Increment, and Data Cut—across MNIST, CIFAR-10, and smallNORB using a CNN-Capsule architecture and reports up to $50\%$ runtime reductions with dataset-dependent effects on accuracy. The findings show that targeted data reductions can maintain or even improve performance in some cases (notably MNIST and certain smallNORB runs) while random or excessive reductions can degrade accuracy, especially on CIFAR-10. The work highlights the potential for substantial efficiency gains in AI training and points to future research on principled data selection to optimize performance-cost tradeoffs.

Abstract

We propose a novel method for training a neural network for image classification to reduce input data dynamically, in order to reduce the costs of training a neural network model. As Deep Learning tasks become more popular, their computational complexity increases, leading to more intricate algorithms and models which have longer runtimes and require more input data. The result is a greater cost on time, hardware, and environmental resources. By using data reduction techniques, we reduce the amount of work performed, and therefore the environmental impact of AI techniques, and with dynamic data reduction we show that accuracy may be maintained while reducing runtime by up to 50%, and reducing carbon emission proportionally.

Maintaining Performance with Less Data

TL;DR

The paper tackles the high cost and environmental impact of training image classifiers by introducing three dynamic data-use methods to reduce input data during training. It evaluates these methods—Data Step, Data Increment, and Data Cut—across MNIST, CIFAR-10, and smallNORB using a CNN-Capsule architecture and reports up to runtime reductions with dataset-dependent effects on accuracy. The findings show that targeted data reductions can maintain or even improve performance in some cases (notably MNIST and certain smallNORB runs) while random or excessive reductions can degrade accuracy, especially on CIFAR-10. The work highlights the potential for substantial efficiency gains in AI training and points to future research on principled data selection to optimize performance-cost tradeoffs.

Abstract

We propose a novel method for training a neural network for image classification to reduce input data dynamically, in order to reduce the costs of training a neural network model. As Deep Learning tasks become more popular, their computational complexity increases, leading to more intricate algorithms and models which have longer runtimes and require more input data. The result is a greater cost on time, hardware, and environmental resources. By using data reduction techniques, we reduce the amount of work performed, and therefore the environmental impact of AI techniques, and with dynamic data reduction we show that accuracy may be maintained while reducing runtime by up to 50%, and reducing carbon emission proportionally.
Paper Structure (20 sections, 1 equation, 8 figures, 11 tables)

This paper contains 20 sections, 1 equation, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Increasing data use
  4. Reducing data use
  5. Variable data use
  6. Methods
  7. Datasets
  8. Hardware
  9. Performance Metrics
  10. Network Architecture
  11. Experimentation
  12. Benchmark
  13. Data Step
  14. Starting with 25% of the dataset
  15. Starting with 50% of the dataset
  16. ...and 5 more sections

Figures (8)

  • Figure 1: Common model structure with Online Augmentation. $\lambda=$ full dataset, $\lambda_a=$ augmented dataset. $f_a()=$ augmentation function.
  • Figure 2: Proposed model structure with Dynamic Data Selection. $\lambda=$ full dataset, $\lambda_r=$ reduced dataset, $\lambda_a=$ augmented dataset. $f_r()=$ reduction function, $f_a()=$ augmentation function
  • Figure 3: Data Step Experiments starting with 25% data. (a) Step at 25% through training, (b) Step at 50% through training, (c) Step at 75% through training
  • Figure 4: Data Step Experiments starting with 50% data. (a) Step at 25% through training, (b) Step at 50% through training, (c) Step at 75% through training
  • Figure 5: Data Step Experiments starting with 75% data. (a) Step at 25% through training, (b) Step at 50% through training, (c) Step at 75% through training
  • ...and 3 more figures