Elastic ViTs from Pretrained Models without Retraining

TL;DR

The paper addresses the challenge of deploying large vision transformers under diverse compute budgets by introducing SnapViT, a post-pretraining structured pruning method that yields elastic subnetworks without retraining or labeled data. It combines a local Hessian proxy derived from self-supervised gradients with a global Hessian proxy learned via Exponential Natural Evolution Strategy to capture cross-block interactions, forming a unified prunability score that enables single-shot pruning across a continuum of sparsities. The approach achieves competitive or superior results across supervised and self-supervised ViTs and semantic segmentation, including large-scale models, while maintaining efficiency (less than five minutes on an A100 for continuum pruning) and enabling post-pruning corrections or full fine-tuning. This work advances practical deployment of vision foundation models by enabling flexible, retraining-free elastic inference with robust performance across datasets and tasks, and it provides public code and pruned models for reproducibility and adoption.

Abstract

Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: Single-shot network approximation for pruned Vision Transformers, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining-free. Experiments on DINO, SigLIPv2, DeIT, and AugReg models demonstrate superior performance over state-of-the-art methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: https://elastic.ashita.nl/

Figures (13)

• Figure 1: Overview of our pruning method. We decompose the Hessian into local and global components: the local Hessian is approximated from self-supervised gradients, while the global Hessian models cross-block correlations learned via an evolutionary algorithm. Combining both yields a unified prunability score that ranks parameters once, allowing single-shot generation of sub-networks at any desired sparsity. The pseudocode for our algorithm is listed in Appendix \ref{['code:pseudo']}.
• Figure 2: Our method matches or improves upon the state-of-the-art in a retraining-free setup, while not using labels. Top-1 accuracies in k-nearest neighbor and linear classification averaged across 7 datasets for supervised AugReg and DeIT ViT-B/16 models. Our label-free method outperforms or matches baselines that utilize labels, especially at high sparsity ratios.
• Figure 3: Our method outperforms baselines for DINO ViT-B/16. Top-1 accuracy in k-nearest neighbor and linear classification averaged across 7 datasets for models pruned with our method, LAMP, and SNIP Magnitude. Our method can prune DINO to 40% sparsity with an accuracy degradation under 5%.
• Figure 4: Large-scale pretraining complicates pruning. Top-1 accuracy in k-nearest neighbor and linear classification for pruned DINOv3 simeoni2025dinov3 and SigLIPv2 ViT-B/16 tschannen2025siglip models. We find that self-supervised models trained on large datasets are harder to prune, benefit from longer optimization horizons, and optimizing for more sparsities.
• Figure 5: Weight correction helps retaining post-pruning performance. A single SparseGPT-style weight correction step greatly improves performance at high sparsity levels while preserving efficiency. Our method matches or surpasses state-of-the-art baselines across pruning ratios and preserves self-supervised model accuracy even under extreme sparsity (bottom row).