Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Impact of Data Duplication on Deep Neural Network-Based Image Classifiers: Robust vs. Standard Models

Alireza Aghabagherloo, Aydin Abadi, Sumanta Sarkar, Vishnu Asutosh Dasu, Bart Preneel

TL;DR

The paper investigates how duplicated training data influences generalization, training efficiency, and robustness of image classifiers, addressing a gap relative to duplication studies in language models. It combines a theoretical bias-variance perspective with empirical analyses on Gaussian data and CIFAR-10, comparing standard and adversarially trained models. Key findings show that non-uniform duplication biases decision boundaries and degrades generalization, while uniform duplication provides at most modest gains; in adversarial settings, duplication more consistently harms accuracy and robustness. The results underscore the importance of deduplication for efficient and reliable image classification, and point to future work on privacy-preserving and federated learning scenarios where duplication is prevalent.

Abstract

The accuracy and robustness of machine learning models against adversarial attacks are significantly influenced by factors such as training data quality, model architecture, the training process, and the deployment environment. In recent years, duplicated data in training sets, especially in language models, has attracted considerable attention. It has been shown that deduplication enhances both training performance and model accuracy in language models. While the importance of data quality in training image classifier Deep Neural Networks (DNNs) is widely recognized, the impact of duplicated images in the training set on model generalization and performance has received little attention. In this paper, we address this gap and provide a comprehensive study on the effect of duplicates in image classification. Our analysis indicates that the presence of duplicated images in the training set not only negatively affects the efficiency of model training but also may result in lower accuracy of the image classifier. This negative impact of duplication on accuracy is particularly evident when duplicated data is non-uniform across classes or when duplication, whether uniform or non-uniform, occurs in the training set of an adversarially trained model. Even when duplicated samples are selected in a uniform way, increasing the amount of duplication does not lead to a significant improvement in accuracy.

Impact of Data Duplication on Deep Neural Network-Based Image Classifiers: Robust vs. Standard Models

TL;DR

The paper investigates how duplicated training data influences generalization, training efficiency, and robustness of image classifiers, addressing a gap relative to duplication studies in language models. It combines a theoretical bias-variance perspective with empirical analyses on Gaussian data and CIFAR-10, comparing standard and adversarially trained models. Key findings show that non-uniform duplication biases decision boundaries and degrades generalization, while uniform duplication provides at most modest gains; in adversarial settings, duplication more consistently harms accuracy and robustness. The results underscore the importance of deduplication for efficient and reliable image classification, and point to future work on privacy-preserving and federated learning scenarios where duplication is prevalent.

Abstract

The accuracy and robustness of machine learning models against adversarial attacks are significantly influenced by factors such as training data quality, model architecture, the training process, and the deployment environment. In recent years, duplicated data in training sets, especially in language models, has attracted considerable attention. It has been shown that deduplication enhances both training performance and model accuracy in language models. While the importance of data quality in training image classifier Deep Neural Networks (DNNs) is widely recognized, the impact of duplicated images in the training set on model generalization and performance has received little attention. In this paper, we address this gap and provide a comprehensive study on the effect of duplicates in image classification. Our analysis indicates that the presence of duplicated images in the training set not only negatively affects the efficiency of model training but also may result in lower accuracy of the image classifier. This negative impact of duplication on accuracy is particularly evident when duplicated data is non-uniform across classes or when duplication, whether uniform or non-uniform, occurs in the training set of an adversarially trained model. Even when duplicated samples are selected in a uniform way, increasing the amount of duplication does not lead to a significant improvement in accuracy.

Paper Structure

This paper contains 15 sections, 4 theorems, 10 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Related Work
  4. Theoretical Perspective on the Impact of Duplicates on Generalization Error
  5. Generalization Error
  6. Generalization Error in Adversarial Settings
  7. Experimental Analysis of the Impact of Duplication in the Training Set
  8. Generalization Error with Gaussian-Distributed Data
  9. Uniform Duplication
  10. Non-uniform Duplication
  11. Generalization Error with CIFAR-10 Dataset
  12. Results and Observations in Standard Model
  13. Results and Observations in Robust Model
  14. Conclusion and Future Work
  15. Theoretical Model: Gaussian Data Generation and SVM Classification

Key Result

Theorem 1

For a prediction model $\hat{f}(x)$ trained on a dataset $D$ to estimate the target function $f(x)$ using an MSE loss function, the bias-variance trade-off is given by:

Figures (4)

  • Figure 1: Comparison of accuracy results (left subfigure) and the percentage of duplications from class +1 to the total number of duplications (right subfigure, denoted as Duplication Ratio for +1 (D-ratio +1)) with varying levels of data duplication (denoted as Duplication Rate (D-rate)) in a uniform duplication setting.
  • Figure 2: Comparison of accuracy results (left subfigure) and the percentage of duplications from class +1 to the whole number of duplications (right subfigure) with varying levels of data duplication in a non-uniform setting.
  • Figure 3: The effect of adding repetitive samples to the training set on test accuracy, the accuracy of the model on the training set including duplicated samples (repetitive accuracy), and the accuracy of the model on the training set excluding duplicated samples (non-repetitive accuracy).
  • Figure 4: Impact of repetitive data in the training set on training accuracy, test accuracy, and the robustness (adversarial accuracy) of an adversarially trained model.

Theorems & Definitions (4)

  • Theorem 1
  • Proposition 1
  • Theorem 2
  • Proposition 2