Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Evaluation of DNN for Past Dataset in Incremental Learning

Naoto Sato

TL;DR

This work tackles the challenge of quickly assessing how incremental updates to a DNN affect accuracy on past datasets in domain-incremental learning. It introduces a gradient-based estimator that computes pre-update gradients and uses the weight change to derive PI and NI, which are combined into an EF score and fed into a linear regression model to predict accuracy changes for past data without exhaustive testing. The method achieves near-constant post-update evaluation time with respect to past dataset size and demonstrates reasonable predictive power (average $R^2$ around 0.6) across MNIST, Fashion-MNIST, and GTSRB, while highlighting limitations in cases with small effect sizes or strong concept drift. Practically, this enables faster, data-efficient decision-making for adopting updated DNNs in real-time systems, with per-class analysis improving reliability in predicting accuracy changes.

Abstract

During the operation of a system including a deep neural network (DNN), new input values that were not included in the training dataset are given to the DNN. In such a case, the DNN may be incrementally trained with the new input values; however, that training may reduce the accuracy of the DNN in regard to the dataset that was previously obtained and used for the past training. It is necessary to evaluate the effect of the additional training on the accuracy for the past dataset. However, evaluation by testing all the input values included in the past dataset takes time. Therefore, we propose a new method to quickly evaluate the effect on the accuracy for the past dataset. In the proposed method, the gradient of the parameter values (such as weight and bias) for the past dataset is extracted by running the DNN before the training. Then, after the training, its effect on the accuracy with respect to the past dataset is calculated from the gradient and update differences of the parameter values. To show the usefulness of the proposed method, we present experimental results with several datasets. The results show that the proposed method can estimate the accuracy change by additional training in a constant time.

Fast Evaluation of DNN for Past Dataset in Incremental Learning

TL;DR

This work tackles the challenge of quickly assessing how incremental updates to a DNN affect accuracy on past datasets in domain-incremental learning. It introduces a gradient-based estimator that computes pre-update gradients and uses the weight change to derive PI and NI, which are combined into an EF score and fed into a linear regression model to predict accuracy changes for past data without exhaustive testing. The method achieves near-constant post-update evaluation time with respect to past dataset size and demonstrates reasonable predictive power (average around 0.6) across MNIST, Fashion-MNIST, and GTSRB, while highlighting limitations in cases with small effect sizes or strong concept drift. Practically, this enables faster, data-efficient decision-making for adopting updated DNNs in real-time systems, with per-class analysis improving reliability in predicting accuracy changes.

Abstract

During the operation of a system including a deep neural network (DNN), new input values that were not included in the training dataset are given to the DNN. In such a case, the DNN may be incrementally trained with the new input values; however, that training may reduce the accuracy of the DNN in regard to the dataset that was previously obtained and used for the past training. It is necessary to evaluate the effect of the additional training on the accuracy for the past dataset. However, evaluation by testing all the input values included in the past dataset takes time. Therefore, we propose a new method to quickly evaluate the effect on the accuracy for the past dataset. In the proposed method, the gradient of the parameter values (such as weight and bias) for the past dataset is extracted by running the DNN before the training. Then, after the training, its effect on the accuracy with respect to the past dataset is calculated from the gradient and update differences of the parameter values. To show the usefulness of the proposed method, we present experimental results with several datasets. The results show that the proposed method can estimate the accuracy change by additional training in a constant time.
Paper Structure (19 sections, 14 equations, 7 figures)

This paper contains 19 sections, 14 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Deep Neural Networks
  4. Incremental Learning
  5. Problem Statement
  6. Proposed Method
  7. Positive and Negative Gradients
  8. Effect Estimation
  9. Regression model
  10. Mini-batch for PL and NL
  11. Experiment
  12. Setup
  13. Results
  14. Evaluation and Discussion
  15. Validity of the evaluation
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Flow of incremental learning
  • Figure 2: Usage of the proposed method
  • Figure 3: Decrease in $PL$ and its approximate value
  • Figure 4: Regression models of MNIST dataset
  • Figure 5: $R^2$ scores of regression models
  • ...and 2 more figures

Theorems & Definitions (13)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Definition 7
  • Definition 8
  • Definition 9
  • Definition 10
  • ...and 3 more