Transformer Reconstructed with Dynamic Value Attention
Xiaowei Wang
TL;DR
The paper addresses the transformer bottleneck of using a fixed value per head by introducing Dynamic Value Attention (DVA), which assigns per-pair semantic value updates between queries and keys. It develops two realizations, with a cost-efficient revised solution that eliminates the FFN by computing dynamic updates through an efficient elementwise operation, enabling a single-head transformer to achieve rich representations. Empirical results on GPT-2-style models show a 37.6% reduction in training time and improved learning dynamics, suggesting substantial practical gains for training efficient large language models. The work also provides a semantic interpretation of transformers and highlights future directions for integrating DVA as a plug-in module (e.g., LoRA) to tune models under various conditions.
Abstract
Since transformer was firstly published in 2017, several works have been proposed to optimize it. However, the major structure of transformer remains unchanged, ignoring one of its main intrinsic limitations, which is the same static value is used for every query in a head. Transformer itself tries to solve this problem by implementing multi-head attentions, yet the number of heads is limited by complexity. I propose a method to decide a value for each query dynamically, which could cut down all the redundant heads, keeping only one. Consequently, the following feed forward network could be cut down entirely, as each revised embedding has already fetched enough useful values far beyond the context. As a result, a single-head Dynamic Value Attention (DVA) is all you need in a transformer. According to the experiment, DVA may save 37.6% training time than the original transformer meanwhile increasing the learning capability.