A Survey on Efficient Training of Transformers

TL;DR

Transformers demand enormous compute and memory during training, motivating a survey of efficient training techniques. The paper surveys arithmetic/optimization approaches, data selection strategies, memory-saving methods, and hardware-algorithm co-design to accelerate Transformer training. It highlights practical methods such as AdamW, Lion, SAM, Fixup/ReZero/SkipInit, token masking, activation compression, rematerialization, FP8/AMP, and LoRA-based PET, as well as scalable distributed training frameworks and efficient attention implementations. The authors outline future directions including elastic supernets/NAS, on-device training, and standardized benchmarks to drive adoption and cost reduction.

Abstract

Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.

Figures (1)

• Figure 1: Overview of the efficient training of Transformers including computation efficiency, memory efficiency and hardware/algorithm codesign perspectives.