PATCH: Learnable Tile-level Hybrid Sparsity for LLMs

TL;DR

PATCH introduces a novel tile-level hybrid sparsity for LLMs, enabling a continuous 0–50% global sparsity by classifying weight tiles as either dense or 2:4 sparse. By jointly training a tile-level mask with a learnable 2:4 pattern, PATCH achieves higher accuracy than fixed-2:4 pruning while delivering hardware-friendly speedups via STOICC. The approach supports nonuniform sparsity across layers, exhibits robust tile-size behavior, and remains compatible with quantization and low-rank methods in hybrid compression pipelines. Across models from 0.5B to 8B parameters, PATCH delivers consistent accuracy improvements and up to 1.38x end-to-end speedups on consumer-grade GPUs, highlighting its practical potential for efficient LLM deployment.

Abstract

Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overheads, yet existing approaches face challenges: unstructured sparsity, where nonzeros can appear anywhere, preserves accuracy but yields irregular access patterns that prevent GPU acceleration, while semi-structured 2:4 sparsity is hardware-friendly but enforces a rigid 50% pattern that degrades model quality. To bridge this gap, we introduce PATCH, a hybrid sparsity framework that enables a continuous sparsity ratio between 0% and 50%. PATCH partitions weight matrices into tiles, assigning each tile to be either dense or 2:4 sparse via a learnable mask selection mechanism. This design provides fine-grained control over accuracy-acceleration tradeoffs and supports non-uniform sparsity across layers, leading to superior overall quality. Across models from 0.5B to 8B parameters, PATCH consistently narrows the gap to dense accuracy while delivering practical speedups. For instance, on LLaMA-2 7B with an A6000 GPU, PATCH achieves 1.18x-1.38x end-to-end speedup over dense baselines while improving accuracy by 0.37%-2.96% compared to the state-of-the-art 2:4 pruning method, MaskLLM.

Figures (11)

• Figure 1: Illustration of the PATCH learning process for generating tile-level hybrid masks. Each tile is parameterized by a learnable distribution and sampled with Gumbel Softmax to produce $\Tilde{{\bm{M}}}_{\text{tile}}$. The dense probability is expanded and merged with a 2:4 mask $\Tilde{{\bm{M}}}_{2:4}$, which can be fixed or jointly learned during training, yielding $\Tilde{{\bm{M}}}$. The final mask assigns each tile to remain dense or follow the 2:4 pattern, enabling flexible sparsity across the weight matrix.
• Figure 2: Layer-wise sparsity allocation under different global sparsity budgets for various models. PATCH achieves the target global sparsity while flexibly distributing pruning across transformer layers.
• Figure 3: Sparsity distribution across Attention and MLP layers under varying global sparsity budgets in Qwen-2.5 0.5B.
• Figure 4: Sparsity distribution across Attention and MLP layers under varying global sparsity budgets in Gemma-3 1B.
• Figure 5: Sparsity distribution across Attention and MLP layers under varying global sparsity budgets in LLaMA-3.2 1B.