ModHiFi: Identifying High Fidelity predictive components for Model Modification
Dhruva Kashyap, Chaitanya Murti, Pranav K Nayak, Tanay Narshana, Chiranjib Bhattacharyya
TL;DR
ModHiFi introduces Subset Fidelity to identify high-fidelity, small component subsets that preserve predictive performance when parts of a model are modified without access to training data or the loss. It proves a local-to-global bound for Lipschitz networks, enabling a principled, architecture-agnostic approach to pruning and unlearning based on local reconstruction, with singleton components often sufficing. The ModHiFi framework (ModHiFi-P for structured pruning and ModHiFi-U for classwise unlearning) uses distributional access via synthetic data to select HiFi components and apply targeted edits without retraining, achieving strong ImageNet pruning speedups and complete CIFAR-10 unlearning, while remaining competitive on LLM pruning. Empirical results across CNNs, Transformers, and LLMs support the existence and effectiveness of HiFi components and demonstrate practical, data-free model modification with broad applicability and minimal fine-tuning.
Abstract
Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as pruning or unlearning constrained by this unavailability an active area of research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible in settings with limited computational resources. In this work, we investigate the fundamental question of identifying components that are critical to the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. We theoretically demonstrate that the global reconstruction error is linearly bounded by local reconstruction errors for Lipschitz-continuous networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity). This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term Subset Fidelity. In the uncorrelated features setting, selecting individual components via their Subset Fidelity scores is optimal, which we use to propose ModHiFi, an algorithm for model modification that requires no training data or loss function access. ModHiFi-P, for structured pruning, achieves an 11% speedup over the current state of the art on ImageNet models and competitive performance on language models. ModHiFi-U, for classwise unlearning, achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin Transformers.