POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models

TL;DR

POP presents a practical online structural pruning framework that leverages contextual sparsity during autoregressive decoding by partitioning FFN channels into Retained, Candidate, and Pruned regions established at a prefilling pass. During decoding, only the Candidate region is evaluated to produce a context-conditioned mask, enabling fine-grained adaptation with minimal overhead and no offline calibration. Empirical results across dense LLMs, MoEs, and VLMs show POP consistently improves accuracy over state-of-the-art pruning methods while delivering meaningful latency reductions and modest FLOPs overhead. The method’s plug-and-play nature and robust performance across modalities underscore the importance of dynamic pruning decisions aligned with generation context for efficient, scalable inference of large foundation models.

Abstract

Large foundation models (LFMs) achieve strong performance through scaling, yet current structural pruning methods derive fixed pruning decisions during inference, overlooking sparsity patterns that emerge in the autoregressive token generation. In this paper, we propose POP (Partition-guided Online Pruning), an efficient online structural pruning framework that enables context-conditioned dynamic pruning with minimal computational overhead. POP partitions model channels into retained, candidate, and pruned regions, where prefilling defines a coarse pruning partition, and the decoding stage generates a fine-grained mask within the candidate region, avoiding full-channel re-evaluation. The coarse pruning partition preserves consistently important weights, while the fine-grained masking provides context-conditioned variation during decoding. Moreover, POP is a lightweight, plug-and-play method that requires no preprocessing, including offline calibration, retraining, or learning predictors. Extensive evaluations across diverse LFMs, including large language models (LLMs), mixture-of-experts models (MoEs), and vision-language models (VLMs), demonstrate that POP consistently delivers higher accuracy than existing pruning approaches while incurring smaller computational overhead and minimizing inference latency.

Figures (4)

• Figure 1: Task-wise normalized performance across benchmarks on Llama2-7B, together with inference efficiency. POP exhibits more consistent performance across tasks while maintaining higher efficiency than baseline pruning methods.
• Figure 2: Decoding-time structural dynamics under online pruning. (a) Channel-rank evolution across decoding steps, where channels are ordered by their prefilling-stage importance and colors indicate normalized rank percentiles. (b) Mean rank difference and Top-50% channel overlap with respect to the prefilling ranking on the generation benchmarks GSM8K and MBPP using Llama3.1-8B, reflecting the degree of ranking variation and the stability of highly important channels during decoding.
• Figure 3: The overview of Partition-guided Online Pruning (POP). (a) During the prefilling stage, FFN channels in each layer are ranked by importance and partitioned into retained, candidate, and pruned regions, forming a coarse pruning partition. (b) During decoding, POP performs online pruning by dynamically selecting candidate channels at each generation step using step-wise pruning masks. (c) At each decoding step, intermediate activations are used to compute importance scores of candidates and update the pruning mask before the final FFN computation in layer $N$. (d) The proportion of FFN parameters dominates over attention parameters across dense LLMs, MoE LLMs, and VLMs, motivating our FFN-focused pruning. (e) Compared with prior methods, POP requires no offline preparation cost.
• Figure 4: Ablation study on partition fraction $\gamma$ in POP. Left: mean accuracy on zero-shot QA tasks. Right: mean accuracy on generative tasks.