Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules

TL;DR

Variator introduces plug-and-play compression plugins that accelerate pre-trained language models by merging multiple hidden vectors into a single compressed representation, reducing sequence length without altering the backbone. The framework supports dynamic workload adaptation by selecting plugins with different acceleration ratios and incurs minimal parameter overhead due to lightweight compression/decompression layers, trained via a two-step distillation-based process. Across seven NLP benchmarks with backbones like T5, Variator achieves up to $53\%$ FLOPs savings while adding only about $0.9\%$ extra parameters and maintaining accuracy within $<2\%$ of the uncompressed models, with even smaller gaps as model size grows to billions of parameters. The results demonstrate practical, scalable acceleration for real-world deployment and indicate potential extensions to decoder-only LLMs and further neuron-level regularization techniques.

Abstract

Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs. In this paper, we propose Variator, a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins. Compression plugins are designed to reduce the sequence length via compressing multiple hidden vectors into one and trained with original PLMs frozen. Different from traditional model acceleration methods, which compress PLMs to smaller sizes, Variator offers two distinct advantages: (1) In real-world applications, the plug-and-play nature of our compression plugins enables dynamic selection of different compression plugins with varying acceleration ratios based on the current workload. (2) The compression plugin comprises a few compact neural network layers with minimal parameters, significantly saving storage and memory overhead, particularly in scenarios with a growing number of tasks. We validate the effectiveness of Variator on seven datasets. Experimental results show that Variator can save 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%. Moreover, when the model scales to billions of parameters, Variator matches the strong performance of uncompressed PLMs.

Figures (5)

• Figure 1: Illustration of model acceleration with compression plugins.
• Figure 2: Illustration of Variator, which improves the computational efficiency via compressing the hidden vectors.
• Figure 3: Model performance with different compression ratios. The horizontal lines indicate the performance of original PLMs without compression plugins.
• Figure 4: Performance with different backbone sizes.
• Figure 5: The ratio of activated neurons with different compression ratios on two datasets.