Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Self-Supervised Dynamic Incremental Regularised Adaptation

Abanoub Ghobrial, Kerstin Eder

TL;DR

The paper addresses dynamic domain adaptation for autonomous systems, aiming to adapt with few labeled samples while avoiding catastrophic forgetting. It reviews DIRA, which uses elastic weight consolidation with Fisher-based regularisation, and contrasts it with TTT's self-supervised, auxiliary-task approach, highlighting strengths and limitations of each. The authors propose DIRA-SS, a self-supervised extension that inserts an auxiliary task via a multi-head architecture to enable label-free retraining, with plans for experimental validation. If validated, this approach could significantly reduce labeling bottlenecks in real-time, safety-critical adaptive systems by enabling prompt domain adaptation without human-in-the-loop labeling.

Abstract

In this paper, we give an overview of a recently developed method for dynamic domain adaptation, named DIRA, which relies on a few samples in addition to a regularisation approach, named elastic weight consolidation, to achieve state-of-the-art (SOTA) domain adaptation results. DIRA has been previously shown to perform competitively with SOTA unsupervised adaption techniques. However, a limitation of DIRA is that it relies on labels to be provided for the few samples used in adaption. This makes it a supervised technique. In this paper, we propose a modification to the DIRA method to make it self-supervised i.e. remove the need for providing labels. Our proposed approach will be evaluated experimentally in future work.

On Self-Supervised Dynamic Incremental Regularised Adaptation

TL;DR

The paper addresses dynamic domain adaptation for autonomous systems, aiming to adapt with few labeled samples while avoiding catastrophic forgetting. It reviews DIRA, which uses elastic weight consolidation with Fisher-based regularisation, and contrasts it with TTT's self-supervised, auxiliary-task approach, highlighting strengths and limitations of each. The authors propose DIRA-SS, a self-supervised extension that inserts an auxiliary task via a multi-head architecture to enable label-free retraining, with plans for experimental validation. If validated, this approach could significantly reduce labeling bottlenecks in real-time, safety-critical adaptive systems by enabling prompt domain adaptation without human-in-the-loop labeling.

Abstract

In this paper, we give an overview of a recently developed method for dynamic domain adaptation, named DIRA, which relies on a few samples in addition to a regularisation approach, named elastic weight consolidation, to achieve state-of-the-art (SOTA) domain adaptation results. DIRA has been previously shown to perform competitively with SOTA unsupervised adaption techniques. However, a limitation of DIRA is that it relies on labels to be provided for the few samples used in adaption. This makes it a supervised technique. In this paper, we propose a modification to the DIRA method to make it self-supervised i.e. remove the need for providing labels. Our proposed approach will be evaluated experimentally in future work.
Paper Structure (7 sections, 3 equations, 3 figures, 2 tables)

This paper contains 7 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Types of Incremental Learning
  4. Domain Adaptation Frameworks
  5. DIRA Summary
  6. Results Summary
  7. Discussion on self-supervision for DIRA

Figures (3)

  • Figure 1: DIRA (Existing Approach) ghobrial2023dira
  • Figure 2: DIRA-SS, the proposed approach combining features of DIRA and TTT.
  • Figure 3: 3D plot showing variances in ResNet-18 layers across different retrained models on CIFAR10C hendrycks2019benchmarking using DIRA ghobrial2023dira.