HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models

TL;DR

HiPP-Prune is presented, a hierarchical preference-conditioned structured pruning framework that treats pruning as conditional resource allocation under multiple objectives that discovers diverse non-dominated pruning plans and provides controllable robustness--utility trade-offs under matched sparsity budgets.

Abstract

Pruning vision-language models (VLMs) for efficient deployment is challenging because compression can affect not only task utility but also visual grounding, often amplifying object hallucinations even at the same sparsity level. We present HiPP-Prune, a hierarchical preference-conditioned structured pruning framework that treats pruning as conditional resource allocation under multiple objectives. HiPP-Prune makes plan-level decisions: a single policy invocation outputs a global pruning blueprint by factorizing decisions into an overall sparsity budget and a layer-wise allocation, enabling queryable trade-offs via a user-specified preference vector. To account for VLM-specific failure modes, our policy state integrates a visual sensitivity signal derived from attention flow between vision tokens and language hidden states, discouraging over-pruning of vision-critical layers that facilitate cross-modal fusion. We optimize pruning plans with plan-level Group Relative Policy Optimization (GRPO) under a multi-objective return that combines task utility, hallucination robustness (POPE), compression, and a synaptic-flow-inspired stability proxy to reduce unproductive exploration in high-sparsity regimes. Experiments on LLaVA with POPE and ScienceQA demonstrate that HiPP-Prune discovers diverse non-dominated pruning plans and provides controllable robustness--utility trade-offs under matched sparsity budgets.

Figures (2)

• Figure 1: An overview of HiPP-Prune. A pretrained VLM and calibration data are used to construct layer-wise states, integrating activation statistics with vision-aware cross-modal cues. Conditioned on a preference vector, a hierarchical policy generates a one-shot structured pruning plan by factorizing decisions into global sparsity control and layer-wise allocation. Candidate plans are optimized via plan-level GRPO using robustness, utility, and compression feedback, while a SynFlow-inspired stability gate downweights updates from non-viable high-sparsity episodes to stabilize plan search. Finally, the optimized pruning plan is applied to the language backbone, followed by lightweight post-pruning recovery fine-tuning.
• Figure 2: Preference-conditioned operating-point clouds and Pareto navigation. Each point represents a pruning plan generated by a single trained HiPP-Prune policy under varying preference vectors $\mathbf{w}$. Small dots illustrate the breadth of the search space by showing candidate plans sampled during the group-relative optimization process. Large dots denote the final Pareto-optimal blueprints selected by the policy for representative preference queries. Coloring reflects the sampled robustness weight $w_{\text{rob}}$ (ranging from 0 to 1), confirming that the policy effectively navigates the trade-off space between hallucination robustness and task utility by modulating the input vector $\mathbf{w}$.