POP: Prefill-Only Pruning for Efficient Large Model Inference

TL;DR

POP tackles the high inference cost of large models by recognizing stage-specific pruning needs in prefill versus decode. It uses a virtual-gate based layer-importance analysis to identify deep layers as redundant for prefill but essential for next-token prediction, and engineers Prefill-Only Pruning (POP) with independent KV projections and a boundary strategy to prune those layers only during prefill. Across Llama-3.1, Qwen3-VL, and Gemma-3, POP achieves up to $1.37\times$ prefill speedup with minimal accuracy loss, outperforming traditional structured pruning benchmarks and matching unstructured pruning in accuracy while maintaining hardware compatibility. The approach demonstrates that aligning pruning with the functional roles of inference stages can substantially improve efficiency without retraining, with potential for broader application in quantization and system design for disaggregated hardware.

Abstract

Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable capabilities. However, their deployment is hindered by significant computational costs. Existing structured pruning methods, while hardware-efficient, often suffer from significant accuracy degradation. In this paper, we argue that this failure stems from a stage-agnostic pruning approach that overlooks the asymmetric roles between the prefill and decode stages. By introducing a virtual gate mechanism, our importance analysis reveals that deep layers are critical for next-token prediction (decode) but largely redundant for context encoding (prefill). Leveraging this insight, we propose Prefill-Only Pruning (POP), a stage-aware inference strategy that safely omits deep layers during the computationally intensive prefill stage while retaining the full model for the sensitive decode stage. To enable the transition between stages, we introduce independent Key-Value (KV) projections to maintain cache integrity, and a boundary handling strategy to ensure the accuracy of the first generated token. Extensive experiments on Llama-3.1, Qwen3-VL, and Gemma-3 across diverse modalities demonstrate that POP achieves up to 1.37$\times$ speedup in prefill latency with minimal performance loss, effectively overcoming the accuracy-efficiency trade-off limitations of existing structured pruning methods.

Figures (2)

• Figure 1: Importance scores of layers from different models over datasets.
• Figure 2: Cosine similarity of internal states between the POP-pruned model and the full model across deep layers (25-36).