Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RTF-Q: Efficient Unsupervised Domain Adaptation with Retraining-free Quantization

Nanyang Du, Chen Tang, Yuxiao Jiang, Yuan Meng, Zhi Wang

TL;DR

This work uses low-precision quantization architectures with varying computational costs, adapting to devices with dynamic computation budgets and introduces multi-bitwidth joint training and the SandwichQ rule, both of which are effective in handling multiple quantization bit-widths across subnets.

Abstract

Performing unsupervised domain adaptation on resource-constrained edge devices is challenging. Existing research typically adopts architecture optimization (e.g., designing slimmable networks) but requires expensive training costs. Moreover, it does not consider the considerable precision redundancy of parameters and activations. To address these limitations, we propose efficient unsupervised domain adaptation with ReTraining-Free Quantization (RTF-Q). Our approach uses low-precision quantization architectures with varying computational costs, adapting to devices with dynamic computation budgets. We subtly configure subnet dimensions and leverage weight-sharing to optimize multiple architectures within a single set of weights, enabling the use of pre-trained models from open-source repositories. Additionally, we introduce multi-bitwidth joint training and the SandwichQ rule, both of which are effective in handling multiple quantization bit-widths across subnets. Experimental results demonstrate that our network achieves competitive accuracy with state-of-the-art methods across three benchmarks while significantly reducing memory and computational costs.

RTF-Q: Efficient Unsupervised Domain Adaptation with Retraining-free Quantization

TL;DR

This work uses low-precision quantization architectures with varying computational costs, adapting to devices with dynamic computation budgets and introduces multi-bitwidth joint training and the SandwichQ rule, both of which are effective in handling multiple quantization bit-widths across subnets.

Abstract

Performing unsupervised domain adaptation on resource-constrained edge devices is challenging. Existing research typically adopts architecture optimization (e.g., designing slimmable networks) but requires expensive training costs. Moreover, it does not consider the considerable precision redundancy of parameters and activations. To address these limitations, we propose efficient unsupervised domain adaptation with ReTraining-Free Quantization (RTF-Q). Our approach uses low-precision quantization architectures with varying computational costs, adapting to devices with dynamic computation budgets. We subtly configure subnet dimensions and leverage weight-sharing to optimize multiple architectures within a single set of weights, enabling the use of pre-trained models from open-source repositories. Additionally, we introduce multi-bitwidth joint training and the SandwichQ rule, both of which are effective in handling multiple quantization bit-widths across subnets. Experimental results demonstrate that our network achieves competitive accuracy with state-of-the-art methods across three benchmarks while significantly reducing memory and computational costs.
Paper Structure (17 sections, 2 equations, 3 figures, 2 tables)

This paper contains 17 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retraining-Free Domain Adaptation
  4. Efficient Domain Adaptation
  5. Network Quantization
  6. METHOD
  7. Preliminaries
  8. Approach Overview
  9. Quantization-Aware Training
  10. Subnet Dimensions
  11. Joint Training
  12. SandwichQ Rule
  13. Experiments
  14. Experimental Setting
  15. Results and Analysis
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The overall framework of RTF-Q. Left: We first configure subnets employing weight-sharing and low-precision quantization architectures with varying MACs' budgets, forming a quantized network bank. Right: We employ the proposed SandwichQ rule to sample a fixed number of networks and optimize their shared parameters using joint training of multiple bit-widths. Only the student's corresponding subnet that meets the MAC budget is used during inference.
  • Figure 2: The average classification accuracy of RTF-Q and RTF-F on the UDA tasks of Office-31, DomainNet, and Office-Home.
  • Figure 3: Effect of joint training across multiple quantization bit-widths on classification accuracy.