Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Experience Report on Regression-Free Repair of Deep Neural Network Model

Takao Nakagawa, Susumu Tokumoto, Shogo Tokui, Fuyuki Ishikawa

TL;DR

The paper addresses the challenge of updating DNNs in industry while avoiding regressions by presenting a regression-controlled repair approach using NeuRecoverLite. It localizes faulty weights through fault localization and optimizes them with particle swarm optimization to repair misclassifications without regressing already correct cases, demonstrated on a car-image classifier with drift and occlusion scenarios. Key findings show that regression suppression is achievable at the overall accuracy level and sometimes at the instance level, though the degree of success depends on target and drift conditions, and hyperparameter design strongly influences outcomes. The work highlights practical considerations for field deployment, including regression guarantees, data drift, and bias in repair results, and points to future work to broaden applicability and reduce the tuning burden, enabling more reliable DNN updates in industry.

Abstract

Systems based on Deep Neural Networks (DNNs) are increasingly being used in industry. In the process of system operation, DNNs need to be updated in order to improve their performance. When updating DNNs, systems used in companies that require high reliability must have as few regressions as possible. Since the update of DNNs has a data-driven nature, it is difficult to suppress regressions as expected by developers. This paper identifies the requirements for DNN updating in industry and presents a case study using techniques to meet those requirements. In the case study, we worked on satisfying the requirement to update models trained on car images collected in Fujitsu assuming security applications without regression for a specific class. We were able to suppress regression by customizing the objective function based on NeuRecover, a DNN repair technique. Moreover, we discuss some of the challenges identified in the case study.

An Experience Report on Regression-Free Repair of Deep Neural Network Model

TL;DR

The paper addresses the challenge of updating DNNs in industry while avoiding regressions by presenting a regression-controlled repair approach using NeuRecoverLite. It localizes faulty weights through fault localization and optimizes them with particle swarm optimization to repair misclassifications without regressing already correct cases, demonstrated on a car-image classifier with drift and occlusion scenarios. Key findings show that regression suppression is achievable at the overall accuracy level and sometimes at the instance level, though the degree of success depends on target and drift conditions, and hyperparameter design strongly influences outcomes. The work highlights practical considerations for field deployment, including regression guarantees, data drift, and bias in repair results, and points to future work to broaden applicability and reduce the tuning burden, enabling more reliable DNN updates in industry.

Abstract

Systems based on Deep Neural Networks (DNNs) are increasingly being used in industry. In the process of system operation, DNNs need to be updated in order to improve their performance. When updating DNNs, systems used in companies that require high reliability must have as few regressions as possible. Since the update of DNNs has a data-driven nature, it is difficult to suppress regressions as expected by developers. This paper identifies the requirements for DNN updating in industry and presents a case study using techniques to meet those requirements. In the case study, we worked on satisfying the requirement to update models trained on car images collected in Fujitsu assuming security applications without regression for a specific class. We were able to suppress regression by customizing the objective function based on NeuRecover, a DNN repair technique. Moreover, we discuss some of the challenges identified in the case study.

Paper Structure

This paper contains 16 sections, 2 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Regression-controlled Repair Technique: NeuRecoverLite
  3. Fault Localization
  4. Particle Swarm Optimization
  5. Field Requirements of DNN Repair
  6. Case Study
  7. Research Questions
  8. Experiment Setup
  9. Results
  10. Lessons Learned
  11. Guaranteeing no regression on test data is difficult
  12. Repairing large amounts of data without regression is difficult
  13. Difficulty of no regression depends on the target to be repaired
  14. Repair difficulty leads to unintended bias in results
  15. How to select hyperparameters
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Hypothetical images of repair target class
  • Figure 2: Accuracy of the minimum regression cases on test data
  • Figure 3: The number of broken/repaired instances of the minimum regression cases on test data
  • Figure 4: The number of broken/repaired instances for comparing EXP A/B and intact weights
  • Figure 5: The number of broken/repaired instances for comparing # of positive sampling
  • ...and 1 more figures