S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance

TL;DR

A novel attention deployment strategy called Sparsity-aware Head-Parallel Load Balance (S-HPLB) is proposed, which can achieve a $2.88\times$ improvement in average attention computation latency without quality degradation.

Abstract

With the increasing volumes of Large Language Models (LLMs) and the expanding context lengths, attention computation has become a key performance bottleneck in LLM serving. For fast attention computation, recent practices often parallelize the attention heads on multiple GPUs, and also widely adopt attention sparsification to reduce the computation amount -- which selectively computes a subset of attention pairs under a preset sparsity budget. In this paper, we notice that attention heads of an LLM model often exhibit heterogeneous-yet-stable sparsity elasticities, which motivates us to enforce head-adaptive sparsity budgets to attain better efficiency while preserving high inference quality. Yet, from the system aspect, with heterogeneous sparsity levels, attention computation time on different heads would be inconsistent, yielding cross-GPU resource bubbles under head-parallel deployment. To further minimize such bubbles, we propose a novel attention deployment strategy called Sparsity-aware Head-Parallel Load Balance (S-HPLB). Experiments on long-context benchmark show that, S-HPLB can achieve a $2.88\times$ improvement in average attention computation latency without quality degradation.

Figures (11)

• Figure 1: Illustration of the attention head parallelism.
• Figure 2: Illustration of the sparse attention workflow.
• Figure 3: Heterogeneity of attention weight recovery ratio across different heads.
• Figure 4: Normalized budget for each head across multiple layers.
• Figure 5: Architecture of S-HPLB.