FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation

TL;DR

FineScope tackles domain-specific adaptation of large language models under computational constraints by jointly optimizing data selection and model compression. It uses sparse autoencoders trained on intermediate activations to curate a domain-aligned dataset, then applies structured pruning guided by this data followed by modified self-distillation to recover domain-specific representations. The results show that pruned models with SAE curated data outperform baselines and can rival larger models on domain tasks, with pruning robustness supported by optimal Top-K settings. The framework demonstrates a practical path to deploy efficient, domain-focused LLMs in real-world, resource-limited environments.

Abstract

Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA, llama} have served as starting points for domain-specific adaptation, but they often suffer from accuracy degradation when tested on specialized datasets. We introduce FineScope, a framework for deriving compact, domain-optimized LLMs from larger pretrained models. FineScope leverages the Sparse Autoencoder (SAE) framework, inspired by its ability to produce interpretable feature representations, to extract domain-specific subsets from large datasets. We apply structured pruning with domain-specific constraints, ensuring that the resulting pruned models retain essential knowledge for the target domain. To further enhance performance, these pruned models undergo self-data distillation, leveraging SAE-curated datasets to restore key domain-specific information lost during pruning. Extensive experiments and ablation studies demonstrate that FineScope achieves highly competitive performance, outperforming several large-scale state-of-the-art LLMs in domain-specific tasks. Additionally, our results show that FineScope enables pruned models to regain a substantial portion of their original performance when fine-tuned with SAE-curated datasets. Furthermore, applying these datasets to fine-tune pretrained LLMs without pruning also improves their domain-specific accuracy, highlighting the robustness of our approach.

Figures (9)

• Figure 1: Overall accuracy (with different model pruning ratio) on the STEM, Social Sciences, and Humanities domains after fine-tuning with Self-Instruct dataset selfinstruct and FineScope dataset. Despite its smaller size, FineScope sustains higher accuracy under aggressive pruning, underscoring its data efficiency and quality.
• Figure 2: Overview of the FineScope : (1) Dataset Curation: (a) Sparse Autoencoder (SAE) is trained on the top-$K$ activations of a pretrained LLM and then used it to extract embedding from datasets,(b)Domain-specific dataset is curated by computing cosine similarity between target domain and the samples in the larger dataset. (2) Pruning and Fine-Tuning: (c) Structured pruning is done w.r.t. selected dataset; (d) Fine-tuned the pruned model using modified self distillation. Here, $U$ denotes the larger dataset, while $D_t$ represents the target domain dataset. The corresponding embeddings are denoted by $e_u$ and $e_t$ respectively.
• Figure 3: Impact of varying TopK on SAE's average reconstruction loss, average accuracy and training time for all transformer blocks.
• Figure 4: Effect of model pruning ratio on accuracy.
• Figure 5: Performance comparison with synthetic STEM dataset.