TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs

TL;DR

TrimLLM introduces progressive layer dropping during domain-focused fine-tuning to compress LLMs without hardware-specific support. By exploiting layer-wide specialization—where MLPs encode domain knowledge and attention handles general semantics—TrimLLM drops less important layers using calibration-scored targets and activation-norm metrics, coupled with sparse updates to minimize forgetting. The approach delivers substantial deployment-time benefits, achieving $2.1-5.7\times$ speedups on consumer GPUs and up to $3.1\times$ on A100 at 50-60% compression, often without accuracy loss, and remains orthogonal to PTQ/pruning, enabling combination with other compression techniques to reach larger speedups. Across medical, legal, and financial domains, TrimLLM demonstrates robust domain specialization with flexible hardware trade-offs, broadening practical LLM deployment without specialized kernels.

Abstract

Specializing large language models (LLMs) for local deployment in domain-specific use cases is necessary for strong performance while meeting latency and privacy constraints. However, conventional task-specific adaptation approaches do not show simultaneous memory saving and inference speedup at deployment time. Practical compression techniques like quantization and pruning require dedicated hardware or kernel support to achieve measured inference speedup. We develop TrimLLM based on the layer-wise specialization phenomenon we empirically observed and verified on contemporary LLMs. TrimLLM reduces the depth of LLMs via progressive layer dropping. We show it retains LLMs' capacity in specific domains and achieves inference speedup irrespective of hardware and deep learning frameworks. We evaluated TrimLLM on LLMs of various sizes for inference; models adapted on medical, legal, and financial datasets all demonstrate $2.1-5.7\times$ inference speedup on consumer GPUs and up to $3.1\times$ speedup on A100 when compared to state-of-the-art model compression algorithms, with no loss in accuracy at 50$\sim$60\% model compression ratio.

Figures (4)

• Figure 1: On SciQ and MedMCQA, LLaMA-7B can be reduced to $40\%\sim 50\%$ of its original size with nearly no loss in accuracy. The layer dropping strategy employed is with calibration scanning, activation-norm tie breaker, and sparse udpate at $r=\frac{1}{4}$.
• Figure 2: The Pareto Frontier of LLaMA-7B-TrimLLM on SciQ and MedMCQA. TrimLLM has a much wider spectrum of operating points to fit the model into different hardware with competitive performance. The layer dropping strategy employed is with calibration scanning and activation-norm tie breaker and + sparse udpate at $r=\frac{1}{4}$.
• Figure 3: Layer dropping patterns when TrimLLM (calibration + activation-norm tie breaker) is applied to LLaMA-7B on QA benchmarks. Results for the first 32 iterations are shown. At this point, the model has been reduced to one half of its original size with nearly no performance loss, evidenced in Figure \ref{['fig:layer_dropping_teaser']}. The numerical value -1 is assigned to discarded layers as accuracy no longer applies.
• Figure 4: An illustration of fine-tuning time complexity for different combinations of sparse fine-tuning schemes and number of layers to be dropped versus full fine-tuning. For all curves drawn, we use normalized $c=1$, $N=64$ to simulate the total number of layers in LLaMA-7B. When two layers are dropped per iteration with sparse FT at $r=\frac{1}{4}$, it only requires 5 epochs of fine-tuning to achieve 50% model compression ratio.

Theorems & Definitions (1)

• Proposition 1