Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DIRA: Dynamic Domain Incremental Regularised Adaptation

Abanoub Ghobrial, Xuan Zheng, Darryl Hond, Hamid Asgari, Kerstin Eder

TL;DR

Dynamic Incremental Regularised Adaptation (DIRA), an approach for dynamic operational domain adaption of DNNs using regularisation techniques, is introduced and shows that DIRA improves on the problem of forgetting and achieves strong gains in performance when retraining using a few samples from the target domain.

Abstract

Autonomous systems (AS) often use Deep Neural Network (DNN) classifiers to allow them to operate in complex, high-dimensional, non-linear, and dynamically changing environments. Due to the complexity of these environments, DNN classifiers may output misclassifications during operation when they face domains not identified during development. Removing a system from operation for retraining becomes impractical as the number of such AS increases. To increase AS reliability and overcome this limitation, DNN classifiers need to have the ability to adapt during operation when faced with different operational domains using a few samples (e.g. 2 to 100 samples). However, retraining DNNs on a few samples is known to cause catastrophic forgetting and poor generalisation. In this paper, we introduce Dynamic Incremental Regularised Adaptation (DIRA), an approach for dynamic operational domain adaption of DNNs using regularisation techniques. We show that DIRA improves on the problem of forgetting and achieves strong gains in performance when retraining using a few samples from the target domain. Our approach shows improvements on different image classification benchmarks aimed at evaluating robustness to distribution shifts (e.g.CIFAR-10C/100C, ImageNet-C), and produces state-of-the-art performance in comparison with other methods from the literature.

DIRA: Dynamic Domain Incremental Regularised Adaptation

TL;DR

Dynamic Incremental Regularised Adaptation (DIRA), an approach for dynamic operational domain adaption of DNNs using regularisation techniques, is introduced and shows that DIRA improves on the problem of forgetting and achieves strong gains in performance when retraining using a few samples from the target domain.

Abstract

Autonomous systems (AS) often use Deep Neural Network (DNN) classifiers to allow them to operate in complex, high-dimensional, non-linear, and dynamically changing environments. Due to the complexity of these environments, DNN classifiers may output misclassifications during operation when they face domains not identified during development. Removing a system from operation for retraining becomes impractical as the number of such AS increases. To increase AS reliability and overcome this limitation, DNN classifiers need to have the ability to adapt during operation when faced with different operational domains using a few samples (e.g. 2 to 100 samples). However, retraining DNNs on a few samples is known to cause catastrophic forgetting and poor generalisation. In this paper, we introduce Dynamic Incremental Regularised Adaptation (DIRA), an approach for dynamic operational domain adaption of DNNs using regularisation techniques. We show that DIRA improves on the problem of forgetting and achieves strong gains in performance when retraining using a few samples from the target domain. Our approach shows improvements on different image classification benchmarks aimed at evaluating robustness to distribution shifts (e.g.CIFAR-10C/100C, ImageNet-C), and produces state-of-the-art performance in comparison with other methods from the literature.
Paper Structure (18 sections, 9 equations, 2 figures, 4 tables)

This paper contains 18 sections, 9 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Types of Incremental Learning
  4. Domain Adaptation Frameworks
  5. Regularisation
  6. Method
  7. Elastic Weight Consolidation
  8. Dynamic Incremental Regularised Adaptation (DIRA)
  9. Experimentation Setup
  10. Benchmarks
  11. Baselines
  12. Models and Hardware
  13. Optimisation Details
  14. Results & Discussion
  15. Regularisation effect on adaptation
  16. ...and 3 more sections

Figures (2)

  • Figure 1: ResNet-26 mean classification accuracy over 15 different corruption types on CIFAR-10C at the highest severity (Level 5).
  • Figure 2: Dynamic adaptation scenario example for DIRA to different domains from CIFAR-10C. Pre-trained ResNet-26 on CIFAR-10 adapts to different corruption examples from CIFAR-10C dataset at the highest severity (Level 5), to show how well DIRA can dynamically adapt to operational domains.