READ: Recurrent Adaptation of Large Transformers

TL;DR

This work tackles the high resource cost of fine-tuning large transformers by introducing READ, a memory-efficient fine-tuning approach that pairs a frozen backbone with a lightweight recurrent side network. READ learns corrections to backbone layer representations via a small RNN and a Joiner, avoiding backpropagation through the large backbone and achieving substantial energy and memory savings while maintaining or improving task performance on GLUE. The method scales independently of backbone size, outperforming several PETL baselines and offering competitive results with significantly reduced training cost. This approach broadens the practicality of adapting large foundation models in resource-constrained environments, with implications for rapid deployment and sustainability in AI systems.

Abstract

Fine-tuning large-scale Transformers has led to the explosion of many AI applications across Natural Language Processing and Computer Vision tasks. However, fine-tuning all pre-trained model parameters becomes impractical as the model size and number of tasks increase. Parameter-efficient transfer learning (PETL) methods aim to address these challenges. While effective in reducing the number of trainable parameters, PETL methods still require significant energy and computational resources to fine-tune. In this paper, we introduce \textbf{RE}current \textbf{AD}aption (READ) -- a lightweight and memory-efficient fine-tuning method -- to overcome the limitations of the current PETL approaches. Specifically, READ inserts a small RNN network alongside the backbone model so that the model does not have to back-propagate through the large backbone network. Through comprehensive empirical evaluation of the GLUE benchmark, we demonstrate READ can achieve a $56\%$ reduction in the training memory consumption and an $84\%$ reduction in the GPU energy usage while retraining high model quality compared to full-tuning. Additionally, the model size of READ does not grow with the backbone model size, making it a highly scalable solution for fine-tuning large Transformers.

Figures (5)

• Figure 1: (Left) Comparison of READ and other fine-tuning methods over GLUE tasks on training energy. (Center) Peak training memory relative to full-tuning. (Right) Normalized energy consumption relative to full-tuning on GLUE tasks.
• Figure 2: READ Fine-Tuning Mechanism.
• Figure 3: Commuting diagram for correction.
• Figure 4: The number of trainable parameters as the backbone model size increases.
• Figure 5: Inference latency as backbone model size increases.

Theorems & Definitions (1)

• Definition 2.1: Correction