Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

TL;DR

This work tackles the computational and environmental costs of code-generating LLMs by introducing Flab-Pruner, a unified structural pruning framework that simultaneously prunes vocabulary, layers, and FFN components. The pruning objective minimizes the KL divergence between the original and pruned models' output distributions to preserve code-generation quality, while a code-specific instruction-tuning strategy accelerates performance recovery after pruning. Empirical results across CodeQwen variants on code generation, chain-of-thought generation, and output prediction show that roughly 22% parameter pruning yields about 97% of the original performance, with post-training reclaiming or surpassing the dense model and delivering substantial gains in storage, GPU usage, FLOPs, and CO2 emissions. The approach demonstrates practical viability for green software engineering, supported by ablations, robustness tests, and a comprehensive CodeHarmony/BigCodeBench evaluation, and is complemented by an open-source release to facilitate real-world deployment and further research.

Abstract

The extensive application of Large Language Models (LLMs) in generative coding tasks has raised concerns due to their high computational demands and energy consumption. Unlike previous structural pruning methods designed for classification models that deal with lowdimensional classification logits, generative Code LLMs produce high-dimensional token logit sequences, making traditional pruning objectives inherently limited. Moreover, existing single component pruning approaches further constrain the effectiveness when applied to generative Code LLMs. In response, we propose Flab-Pruner, an innovative unified structural pruning method that combines vocabulary, layer, and Feed-Forward Network (FFN) pruning. This approach effectively reduces model parameters while maintaining performance. Additionally, we introduce a customized code instruction data strategy for coding tasks to enhance the performance recovery efficiency of the pruned model. Through extensive evaluations on three state-of-the-art Code LLMs across multiple generative coding tasks, the results demonstrate that Flab-Pruner retains 97% of the original performance after pruning 22% of the parameters and achieves the same or even better performance after post-training. The pruned models exhibit significant improvements in storage, GPU usage, computational efficiency, and environmental impact, while maintaining well robustness. Our research provides a sustainable solution for green software engineering and promotes the efficient deployment of LLMs in real-world generative coding intelligence applications.

Figures (2)

• Figure 1: The framework of Flab-Pruner
• Figure 2: Performance comparison of different layer pruning models with different number of layers pruned.