Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression

Zeli Su, Ziyin Zhang, Wenzheng Zhang, Zhou Liu, Guixian Xu, Wentao Zhang

TL;DR

<3-5 sentence high-level summary> SHRP targets encoder efficiency by reframing attention heads as independent experts connected to a shared Expander FFN, enabling joint pruning of attention and FFN components. It employs a progressive, layer-wise MoE-inspired training regime with a two-stage optimization (load balancing then specialization) and a Top-1 routing strategy to produce a deterministic, routing-free encoder after pruning. Empirical results on GLUE with BERT-base show strong accuracy retention even under aggressive compression, accompanied by substantial parameter, FLOP, and throughput gains that are suitable for real-world web-scale deployment. The work demonstrates deployment-oriented structured compression that outperforms traditional head-pruning methods and provides interpretable head specialization patterns through usage statistics.

Abstract

Transformer encoders are widely deployed in large-scale web services for natural language understanding tasks such as text classification, semantic retrieval, and content ranking. However, their high inference latency and memory consumption pose significant challenges for real-time serving and scalability. These limitations stem largely from architectural redundancy, particularly in the attention module. The inherent parameter redundancy of the attention mechanism, coupled with the fact that its attention heads operate with a degree of independence, makes it particularly amenable to structured model compression. In this paper, we propose SHRP (Specialized Head Routing and Pruning), a novel structured pruning framework that automatically identifies and removes redundant attention heads while preserving most of the model's accuracy and compatibility. SHRP introduces Expert Attention, a modular design that treats each attention head as an independent expert, followed by a lightweight shared expander feed-forward network that refines their outputs. The framework employs a unified Top-1 usage-driven mechanism to jointly perform dynamic routing during training and deterministic pruning at deployment. Experimental results on the GLUE benchmark using a BERT-base encoder show that SHRP achieves 93% of the original model accuracy while reducing parameters by 48 percent. Under an extreme compression scenario where 11/12 of the layers are pruned, the model still maintains 84% accuracy and delivers a 4.2x throughput gain while reducing computation to as low as 11.5 percent of the original FLOPs, demonstrating its practical utility for large-scale and latency-sensitive web deployments.

SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression

TL;DR

<3-5 sentence high-level summary> SHRP targets encoder efficiency by reframing attention heads as independent experts connected to a shared Expander FFN, enabling joint pruning of attention and FFN components. It employs a progressive, layer-wise MoE-inspired training regime with a two-stage optimization (load balancing then specialization) and a Top-1 routing strategy to produce a deterministic, routing-free encoder after pruning. Empirical results on GLUE with BERT-base show strong accuracy retention even under aggressive compression, accompanied by substantial parameter, FLOP, and throughput gains that are suitable for real-world web-scale deployment. The work demonstrates deployment-oriented structured compression that outperforms traditional head-pruning methods and provides interpretable head specialization patterns through usage statistics.

Abstract

Transformer encoders are widely deployed in large-scale web services for natural language understanding tasks such as text classification, semantic retrieval, and content ranking. However, their high inference latency and memory consumption pose significant challenges for real-time serving and scalability. These limitations stem largely from architectural redundancy, particularly in the attention module. The inherent parameter redundancy of the attention mechanism, coupled with the fact that its attention heads operate with a degree of independence, makes it particularly amenable to structured model compression. In this paper, we propose SHRP (Specialized Head Routing and Pruning), a novel structured pruning framework that automatically identifies and removes redundant attention heads while preserving most of the model's accuracy and compatibility. SHRP introduces Expert Attention, a modular design that treats each attention head as an independent expert, followed by a lightweight shared expander feed-forward network that refines their outputs. The framework employs a unified Top-1 usage-driven mechanism to jointly perform dynamic routing during training and deterministic pruning at deployment. Experimental results on the GLUE benchmark using a BERT-base encoder show that SHRP achieves 93% of the original model accuracy while reducing parameters by 48 percent. Under an extreme compression scenario where 11/12 of the layers are pruned, the model still maintains 84% accuracy and delivers a 4.2x throughput gain while reducing computation to as low as 11.5 percent of the original FLOPs, demonstrating its practical utility for large-scale and latency-sensitive web deployments.
Paper Structure (28 sections, 7 equations, 2 figures, 5 tables)

This paper contains 28 sections, 7 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Attention Head Pruning.
  4. SHRP Framework
  5. Overview.
  6. From Standard Attention to Expert Attention
  7. Attention as Expert Module
  8. Expander FFN
  9. Top-1 Gating
  10. Integration and Output
  11. Progressive Training for Expert Specialization
  12. Layer-wise MoE Conversion.
  13. Two-Stage Learning: Load Balancing $\rightarrow$ Specialization.
  14. Top-1 Pruning
  15. Usage-based Expert Selection.
  16. ...and 13 more sections

Figures (2)

  • Figure 1: SHRP framework with progressive training, usage analysis, and Top-1 pruning.The framework consists of three key stages: (1) Training, where standard multi-head attention is progressively transformed into Expert Attention via layer-wise conversion; training proceeds in two phases—Balanced Exploration (with load-balancing loss) and Expert Specialization (with Top-1 routing); (2) Head-Usage Analysis, where expert usage is measured per layer based on routing decisions during inference; (3) Top-1 Pruning, where underused experts and routers are removed to produce a compact, deterministic encoder. Each expert head routes through a shared lightweight Expander FFN, enabling efficient structured compression across attention and FFN components.
  • Figure 2: Task-wise visualization of pruning impact across the GLUE benchmark. Each subplot corresponds to a task, showing the effect of SHRP pruning at different layer ratios (2/12 to 11/12). The bar chart (left Y-axis) represents the change in task accuracy compared to the baseline: upward green bars indicate improved performance post-pruning, while downward red bars reflect performance drops. The purple line (right Y-axis) shows the corresponding throughput gain (x-fold), measured without the gating router overhead. The final panel aggregates results across all tasks, highlighting the average performance change and overall efficiency gains.