Mugi: Value Level Parallelism For Efficient LLMs

TL;DR

This work expands value level parallelism (VLP) from large-batch GEMM to the nonlinear operations and small-batch GEMMs dominant in transformer-based LLMs by introducing Mugi, a unified VLP architecture. It formulates a value-centric nonlinear approximation and optimizes asymmetric, small-batch GEMM through grouping attention, weight-only quantization, and KV-cache quantization, reusing the same compute array for both nonlinear and GEMM tasks. The results show substantial gains in throughput and energy efficiency (up to 45× and up to 668× for nonlinear softmax-related tasks; up to 2.07× throughput and 3.11× energy efficiency for LLM workloads) and meaningful reductions in operational and embodied carbon (approximately 1.45× and 1.48×). These findings demonstrate a practical, scalable path to more sustainable, high-performance LLM inference by tightly integrating nonlinear approximation with GEMM acceleration in a single architecture.

Abstract

Value level parallelism (VLP) has been proposed to improve the efficiency of large-batch, low-precision general matrix multiply (GEMM) between symmetric activations and weights. In transformer based large language models (LLMs), there exist more sophisticated operations beyond activation-weight GEMM. In this paper, we explore how VLP benefits LLMs. First, we generalize VLP for nonlinear approximations, outperforming existing nonlinear approximations in end-to-end LLM accuracy, performance, and efficiency. Our VLP approximation follows a value-centric approach, where important values are assigned with greater accuracy. Second, we optimize VLP for small-batch GEMMs with asymmetric inputs efficiently, which leverages timely LLM optimizations, including weight-only quantization, key-value (KV) cache quantization, and group query attention. Finally, we design a new VLP architecture, Mugi, to encapsulate the innovations above and support full LLM workloads, while providing better performance, efficiency and sustainability. Our experimental results show that Mugi can offer significant improvements on throughput and energy efficiency, up to $45\times$ and $668\times$ for nonlinear softmax operations, and $2.07\times$ and $3.11\times$ for LLMs, and also decrease operational carbon for LLM operation by $1.45\times$ and embodied carbon by $1.48\times$.

Figures (17)

• Figure 1: Challenges for LLM inference using VLP.
• Figure 2: Illustration of VLP, detailed in Section \ref{['sec:vlp_review']}.
• Figure 3: VLP approximation for nonlinear operations, exp here, with a floating-point input $i$, represented as S-M-E, denoting the sign, mantissa and exponent. More details are in Section \ref{['subsec:formulation']}.
• Figure 4: Distribution of input values and exponents of nonlinear operations in transformer models. Profiled layers, stages, and sequence lengths are detailed in Table \ref{['tab:studied_llms']}. Cooler colors represent early layers, while warmer colors represent later layers. Within each color, lighter lines represent shorter sequence lengths, while darker lines represent longer sequence lengths. Softmax, SiLU, and GELU are abbreviated as SM, S, and G, denoting the nonlinear function within each window.
• Figure 5: An example sliding window for input mapping. This example chooses the exponent range of $[-3, 4]$ for the current set of inputs, from the full LUT window with the exponent range of $[-6, 5]$. The sliding window size of 8 is chosen to match the VLP array width in prior VLP works carat_paper. This sliding window can slide left and right for each mapping, aiming to minimize the accuracy loss.