SAP: Syntactic Attention Pruning for Transformer-based Language Models

TL;DR

The paper tackles the problem of pruning attention heads in Transformer-based language models without retraining while preserving predictive performance. It introduces Syntactic Attention Pruning (SAP), which uses syntactic dependency statistics and attention patterns to identify and retain heads critical for syntax, supplemented by Candidate Filtering (CF) to guard against performance loss. Empirical results on GLUE show that SAP, especially with CF, preserves important syntactic heads and yields superior retrain-free performance compared to a state-of-the-art math-based pruning method, with improved interpretability via attention-map analysis. The work offers a linguistically grounded, scalable approach to model compression with practical impact for deploying transformer models more efficiently.

Abstract

This paper introduces Syntactic Attention Pruning (SAP), a novel method for effectively pruning attention heads in Transformer models. Unlike conventional approaches that rely solely on mathematical analysis of model weights and activations, SAP incorporates both the syntactic structure and attention patterns of sentences to guide the pruning process. By leveraging these linguistic features, SAP not only achieves performance comparable to state-of-the-art methods but also enhances the interpretability of model behavior. To further improve robustness, we propose Candidate Filtering (CF), a mechanism that prioritizes heads based on their contribution to model performance, mitigating degradation during pruning. Experimental results indicate that SAP effectively preserves critical heads of a high density of strong attention values, outperforming existing head pruning strategies in retrain-free settings. These findings position SAP as a promising foundation for a new direction in model compression research, offering high flexibility for pruning across all transformer-based language models.

Figures (8)

• Figure 1: An illustration of syntactic dependencies.
• Figure 2: Flow of Syntactic Attention Pruning.
• Figure 3: Example of SAP. The sentence is shown with partial syntactic dependencies for brevity.
• Figure 4: Statistic of top-5 syntactic dependencies in GLUE.
• Figure 5: Performance comparison at different sparsity levels.