Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continual Knowledge Consolidation LORA for Domain Incremental Learning

Naeem Paeedeh, Mahardhika Pratama, Weiping Ding, Jimmy Cao, Wolfgang Mayer, Ryszard Kowalczyk

TL;DR

ConEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to embrace domain-specific knowledge and integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications.

Abstract

Domain Incremental Learning (DIL) is a continual learning sub-branch that aims to address never-ending arrivals of new domains without catastrophic forgetting problems. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, existing works create task-specific LoRAs overlooking shared knowledge across tasks. Inaccurate selection of task-specific LORAs during inference results in significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to embrace domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inferences and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to take advantage of intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior arts in 4 popular benchmark problems with over 5% margins.

Continual Knowledge Consolidation LORA for Domain Incremental Learning

TL;DR

ConEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to embrace domain-specific knowledge and integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications.

Abstract

Domain Incremental Learning (DIL) is a continual learning sub-branch that aims to address never-ending arrivals of new domains without catastrophic forgetting problems. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, existing works create task-specific LoRAs overlooking shared knowledge across tasks. Inaccurate selection of task-specific LORAs during inference results in significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to embrace domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inferences and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to take advantage of intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior arts in 4 popular benchmark problems with over 5% margins.
Paper Structure (21 sections, 21 equations, 3 figures, 4 tables, 3 algorithms)

This paper contains 21 sections, 21 equations, 3 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Problem Definition
  5. Low-Rank Adaptation (LoRA)
  6. Stochastic Classifier
  7. Methodology
  8. Network Architecture
  9. Loss Function
  10. Auxiliary Domain Classifiers
  11. Time Complexity Analysis
  12. Experiments
  13. Datasets
  14. Baseline Methods
  15. Implementation Details
  16. ...and 6 more sections

Figures (3)

  • Figure 1: A stochastic classifier. a) The decision boundary for a single cosine classifier with two weight vectors for two classes. The feature vector belongs to the blue class. b) The illustration of the effect of having two weight vectors per class, resulting in four decision boundaries. c) It shows the effect of having an infinite number of weight vectors on the decision boundary. d) The stochastic nature of the classifier improves the feature separation of the model and classifier by resolving the confusion when the margin is unclear.
  • Figure 2: The architecture of the model and pipeline of CONEC-LoRA. CONEC-LoRA consists of two main stages. First, the input is given to the frozen backbone. The logits of the domain classifiers (DCs) are being calculated. The domain is detected based on the decision of the most confident domain classifier. Second, the domain-specific LoRAs are chosen for use in the domain-specific blocks. The gray rectangles represent the frozen layers of the ViT.
  • Figure 3: UMAP plots on the embedding space of the model before training on the left and after training on the right on five domains of the CDDB-Hard dataset.