Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Performance of Analog Training for Transfer Learning

Omobayode Fagbohungbe, Corey Lammie, Malte J. Rasch, Takashi Ando, Tayfun Gokmen, Vijay Narayanan

TL;DR

Addressing the challenge of training neural networks on analog in-memory computing hardware for transfer learning, the paper studies the chopped TTv2 (c-TTv2) algorithm's effectiveness under realistic device non-idealities. It evaluates c-TTv2 with a Swin-ViT model pre-trained on CIFAR10 and fine-tuned on subsets of CIFAR100, employing a one-time weight-transfer noise modeled as $W_{noise}=W_{pre_trained} + τ N(0,1)$. Using IBM aihwkit to simulate HfOx ReRAM, the study explores robustness to weight-transfer noise, symmetric-point skew/variability, pulse update noise, and mean pulse DtoD variability, and compares to analog-from-scratch and digital TL baselines. The results show that c-TTv2 yields competitive or superior performance to digital TL and outperforms analog TL trained from scratch, with robustness that improves as task complexity grows; these findings support the viability of energy-efficient analog TL and guide future work toward larger models and benchmarks.

Abstract

Analog in-memory computing is a next-generation computing paradigm that promises fast, parallel, and energy-efficient deep learning training and transfer learning (TL). However, achieving this promise has remained elusive due to a lack of suitable training algorithms. Analog memory devices exhibit asymmetric and non-linear switching behavior in addition to device-to-device variation, meaning that most, if not all, of the current off-the-shelf training algorithms cannot achieve good training outcomes. Also, recently introduced algorithms have enjoyed limited attention, as they require bi-directionally switching devices of unrealistically high symmetry and precision and are highly sensitive. A new algorithm chopped TTv2 (c-TTv2), has been introduced, which leverages the chopped technique to address many of the challenges mentioned above. In this paper, we assess the performance of the c-TTv2 algorithm for analog TL using a Swin-ViT model on a subset of the CIFAR100 dataset. We also investigate the robustness of our algorithm to changes in some device specifications, including weight transfer noise, symmetry point skew, and symmetry point variability

Assessing the Performance of Analog Training for Transfer Learning

TL;DR

Addressing the challenge of training neural networks on analog in-memory computing hardware for transfer learning, the paper studies the chopped TTv2 (c-TTv2) algorithm's effectiveness under realistic device non-idealities. It evaluates c-TTv2 with a Swin-ViT model pre-trained on CIFAR10 and fine-tuned on subsets of CIFAR100, employing a one-time weight-transfer noise modeled as . Using IBM aihwkit to simulate HfOx ReRAM, the study explores robustness to weight-transfer noise, symmetric-point skew/variability, pulse update noise, and mean pulse DtoD variability, and compares to analog-from-scratch and digital TL baselines. The results show that c-TTv2 yields competitive or superior performance to digital TL and outperforms analog TL trained from scratch, with robustness that improves as task complexity grows; these findings support the viability of energy-efficient analog TL and guide future work toward larger models and benchmarks.

Abstract

Analog in-memory computing is a next-generation computing paradigm that promises fast, parallel, and energy-efficient deep learning training and transfer learning (TL). However, achieving this promise has remained elusive due to a lack of suitable training algorithms. Analog memory devices exhibit asymmetric and non-linear switching behavior in addition to device-to-device variation, meaning that most, if not all, of the current off-the-shelf training algorithms cannot achieve good training outcomes. Also, recently introduced algorithms have enjoyed limited attention, as they require bi-directionally switching devices of unrealistically high symmetry and precision and are highly sensitive. A new algorithm chopped TTv2 (c-TTv2), has been introduced, which leverages the chopped technique to address many of the challenges mentioned above. In this paper, we assess the performance of the c-TTv2 algorithm for analog TL using a Swin-ViT model on a subset of the CIFAR100 dataset. We also investigate the robustness of our algorithm to changes in some device specifications, including weight transfer noise, symmetry point skew, and symmetry point variability
Paper Structure (2 sections, 1 equation, 1 figure)

This paper contains 2 sections, 1 equation, 1 figure.

Table of Contents

  1. Introduction
  2. Methodology

Figures (1)

  • Figure 1: Key conceptual differences between digital TL and our analog TL implementation. We obtain the best results when the base model is converted from digital to analog and a new analog head is introduced.