How much data do you need? Part 2: Predicting DL class specific training dataset sizes

TL;DR

This paper targets the question of predicting machine learning classification model performance, when taking into account the number of training examples per class and not just the overall number of training examples, and suggests an algorithm which is motivated from special cases of space filling design of experiments.

Abstract

This paper targets the question of predicting machine learning classification model performance, when taking into account the number of training examples per class and not just the overall number of training examples. This leads to the a combinatorial question, which combinations of number of training examples per class should be considered, given a fixed overall training dataset size. In order to solve this question, an algorithm is suggested which is motivated from special cases of space filling design of experiments. The resulting data are modeled using models like powerlaw curves and similar models, extended like generalized linear models i.e. by replacing the overall training dataset size by a parametrized linear combination of the number of training examples per label class. The proposed algorithm has been applied on the CIFAR10 and the EMNIST datasets.

Figures (10)

• Figure 1: Descriptive results for the models fitted to CIFAR10 dataset.
• Figure 2: Test accuracies vs. epochs for different number of training dataset sizes for the CIFAR10 experiment.
• Figure 3: Prediction plots for powerlaw function 0 on the training dataset (left), test dataset (middle) and using forward testing on the test dataset (right) for the CIFAR10 experiment.
• Figure 4: Prediction plots for powerlaw function 1 on the training dataset (left), test dataset (middle) and using forward testing on the test dataset (right) for the CIFAR10 experiment.
• Figure 5: Prediction plots for powerlaw function 2 on the training dataset (left), test dataset (middle) and using forward testing on the test dataset (right).