Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning

Feng Chen

TL;DR

Calibration Side Tuning (CST) targets parameter- and memory-efficient transfer learning for object detection by adapting transformer-inspired fine-tuning techniques to ResNet backbones. CST employs a dedicated side network with Maximal Transition Calibration (MTC) gating to fuse backbone and side features, achieving strong accuracy while keeping added parameters and memory footprint small. Through extensive ablations and comparisons on five datasets, CST outperforms existing PETL methods and offers a favorable trade-off between model complexity and performance. This work demonstrates practical benefits for deploying fine-tuned detectors under resource constraints in diverse settings.

Abstract

Achieving a universally high accuracy in object detection is quite challenging, and the mainstream focus in the industry currently lies on detecting specific classes of objects. However, deploying one or multiple object detection networks requires a certain amount of GPU memory for training and storage capacity for inference. This presents challenges in terms of how to effectively coordinate multiple object detection tasks under resource-constrained conditions. This paper introduces a lightweight fine-tuning strategy called Calibration side tuning, which integrates aspects of adapter tuning and side tuning to adapt the successful techniques employed in transformers for use with ResNet. The Calibration side tuning architecture that incorporates maximal transition calibration, utilizing a small number of additional parameters to enhance network performance while maintaining a smooth training process. Furthermore, this paper has conducted an analysis on multiple fine-tuning strategies and have implemented their application within ResNet, thereby expanding the research on fine-tuning strategies for object detection networks. Besides, this paper carried out extensive experiments using five benchmark datasets. The experimental results demonstrated that this method outperforms other compared state-of-the-art techniques, and a better balance between the complexity and performance of the finetune schemes is achieved.

CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning

TL;DR

Calibration Side Tuning (CST) targets parameter- and memory-efficient transfer learning for object detection by adapting transformer-inspired fine-tuning techniques to ResNet backbones. CST employs a dedicated side network with Maximal Transition Calibration (MTC) gating to fuse backbone and side features, achieving strong accuracy while keeping added parameters and memory footprint small. Through extensive ablations and comparisons on five datasets, CST outperforms existing PETL methods and offers a favorable trade-off between model complexity and performance. This work demonstrates practical benefits for deploying fine-tuned detectors under resource constraints in diverse settings.

Abstract

Achieving a universally high accuracy in object detection is quite challenging, and the mainstream focus in the industry currently lies on detecting specific classes of objects. However, deploying one or multiple object detection networks requires a certain amount of GPU memory for training and storage capacity for inference. This presents challenges in terms of how to effectively coordinate multiple object detection tasks under resource-constrained conditions. This paper introduces a lightweight fine-tuning strategy called Calibration side tuning, which integrates aspects of adapter tuning and side tuning to adapt the successful techniques employed in transformers for use with ResNet. The Calibration side tuning architecture that incorporates maximal transition calibration, utilizing a small number of additional parameters to enhance network performance while maintaining a smooth training process. Furthermore, this paper has conducted an analysis on multiple fine-tuning strategies and have implemented their application within ResNet, thereby expanding the research on fine-tuning strategies for object detection networks. Besides, this paper carried out extensive experiments using five benchmark datasets. The experimental results demonstrated that this method outperforms other compared state-of-the-art techniques, and a better balance between the complexity and performance of the finetune schemes is achieved.
Paper Structure (20 sections, 5 equations, 4 figures, 3 tables)

This paper contains 20 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Parameter-efficient Transfer Learning
  4. Memory-Efficient Training
  5. Proposed Method
  6. From Transformer to ResNet
  7. Bit Tuning
  8. Adapter Tuning
  9. Prompt Tuning
  10. Side Tuning
  11. Calibration side tuning
  12. Ladder calibration
  13. Lightweight design
  14. Experiments
  15. Experimental Settings
  16. ...and 5 more sections

Figures (4)

  • Figure 1: The illustration of VL-Adapter
  • Figure 2: The illustration of VPT
  • Figure 3: The illustration of Ladder Side-Tuning
  • Figure 4: The illustration of Calibration Side-Tuning