Addition is All You Need for Energy-efficient Language Models

TL;DR

This work finds that a floating point multiplier can be approximated by one integer adder with high precision, and proposes the linear-complexity multiplication L-Mul algorithm that approximates floating point number multiplication with integer addition operations.

Abstract

Large neural networks spend most computation on floating point tensor multiplications. In this work, we find that a floating point multiplier can be approximated by one integer adder with high precision. We propose the linear-complexity multiplication L-Mul algorithm that approximates floating point number multiplication with integer addition operations. The new algorithm costs significantly less computation resource than 8-bit floating point multiplication but achieves higher precision. Compared to 8-bit floating point multiplications, the proposed method achieves higher precision but consumes significantly less bit-level computation. Since multiplying floating point numbers requires substantially higher energy compared to integer addition operations, applying the L-Mul operation in tensor processing hardware can potentially reduce 95% energy cost by element-wise floating point tensor multiplications and 80% energy cost of dot products. We calculated the theoretical error expectation of L-Mul, and evaluated the algorithm on a wide range of textual, visual, and symbolic tasks, including natural language understanding, structural reasoning, mathematics, and commonsense question answering. Our numerical analysis experiments agree with the theoretical error estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa outperforms float8_e5m2. Evaluation results on popular benchmarks show that directly applying L-Mul to the attention mechanism is almost lossless. We further show that replacing all floating point multiplications with 3-bit mantissa L-Mul in a transformer model achieves equivalent precision as using float8_e4m3 as accumulation precision in both fine-tuning and inference.

Figures (4)

• Figure 1: 16-bit, 8-bit floating point numbers defined in IEEE 754 and on various hardware for tensor computations, and the 16-bit integer. MSB stands for most significant bit and LSB stands for least significant bit.
• Figure 2: Coparing the process of regular floating-point multiplication and linear-complexity multiplication ($\mathcal{L}$-Mul) between two fp32 numbers. In the inline PTX Assembly code, $1 and $2 are fp32 registers storing inputs while $0 is the fp32 register for output. s1, s2, r0, r1, r2 are unsigned int32 registers storing intermediate results. Note that the assembly program is only for numerical simulation on Nvidia GPUs. The optimal implementation is at the hardware level.
• Figure 3: Mean square errors obtained by different $l(k)$ selections on Llama and Gemma models. The combinations achieving higher precision than fp8_e4m3 are highlighted in read, and those outperforming fp8_e5m2 are underlined. When $k=4$ and $l(k)=3$, the average error of the llama model is slighly lower but very close to fp8_e4m3.
• Figure 4: Comparing the error levels of linear-complexity multiplication ($\mathcal{L}$-Mul) against the number of mantissa bits comparing with 8-bit FP multiplication operation in different formats.